diff --git a/search/search_index.json b/search/search_index.json index 6813272..804c11d 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"The Global Radio Explorer Telescope Welcome to the documentation for the Global Radio Explorer Telescope! In the tabs above, you'll find the documentation for the hardware and software architectures as well as guides for construction and setup. Status OVRO - Up and Running! Dashboard","title":"Home"},{"location":"#the-global-radio-explorer-telescope","text":"Welcome to the documentation for the Global Radio Explorer Telescope! In the tabs above, you'll find the documentation for the hardware and software architectures as well as guides for construction and setup.","title":"The Global Radio Explorer Telescope"},{"location":"#status","text":"OVRO - Up and Running! Dashboard","title":"Status"},{"location":"about/","text":"The Global Radio Telescope is an new low-cost radio telescope designed to be an all-sky monitor for bright radio bursts. Building on the success of STARE2, we will search for fast radio bursts (FRBs) emitted from Galactic magnetars as well as bursts from nearby galaxies. GReX will search down to ten microseconds time resolution, allowing us to find new super giant radio pulses from Milky Way pulsars and study their broadband emission. The proposed instrument will employ ultra-wide band (0.7-2 GHz) feeds coupled to a high performance (receiver temperature 10 K) low noise amplifier (LNA) originally developed for the DSA-110 and DSA-2000 projects. In GReX Phase I (GReX-I), unit systems will be deployed at Owens Valley Radio Observatory (OVRO) and Big Smoky Valley, Nevada. Phase II will expand the array, placing feeds in India, Australia, and elsewhere in order to build up to continuous coverage of nearly 4\u03c0 steradians and to increase our exposure to the Galactic plane. We model the local magnetar population to forecast for GReX, finding the improved sensitivity and increased exposure to the Galactic plane could lead to dozens of FRB-like bursts per year. For more information, read our paper here ! Team Project Scientist - Dr. Liam Connor, PhD Project Engineer - Kiran Shila, MSEE","title":"About"},{"location":"about/#team","text":"Project Scientist - Dr. Liam Connor, PhD Project Engineer - Kiran Shila, MSEE","title":"Team"},{"location":"hardware/assembly/","text":"Assembly Guide TODO! Once we have the complete pacakge, write a step by step guide for assembly and installation","title":"Assembly Guide"},{"location":"hardware/assembly/#assembly-guide","text":"TODO! Once we have the complete pacakge, write a step by step guide for assembly and installation","title":"Assembly Guide"},{"location":"hardware/box/","text":"The Box Here one can find documentation re: the GReX box, including cables, routing, etc. A 3D PDF is available for download here . The BOM is available here Interface Plate The custom interface plate is machined from stock aluminum measuring approximately 12x3x1/8 inches. The drilled hole specifications are shown in the drawing below: Interface Plate Drawing Tips for machining The drilled holes are measured such that the stock size does not need to be precise. This plate will later be used to mark the corresponding holes on the main box, so the outer eight holes could be measured from all four sides. We assume the long edges arrive machined parallel. Note that the eight tapped holes around the perimeter of the piece should first be drilled with a #29 bit. After aligning the plate and marking the bottom of the box, the holes are re-drilled using the specified #18 bit. To adhere the interface plate to the main box, first the cutout must be made with a jig saw. Refer to the drawing below for placement. Note that the \"bottom\" of the box is defined when the lid is facing upwards and the hinge is on the left side. Double check that the internal ground pin is not removed or damaged with the cutout. Box Cutout Drawing Tips for machining After marking the box according to the above drawing, we marked four points: one inside each corner of the cutout, offset 3/8\" from the edge. We drilled 3/4\" holes at each point, which served as entry points for the jigsaw blade (20 TPI, no oscillation). File generously to remove sharp edges. After the cutout is finished, use the interface plate to mark the location of the eight screw holes. When determining the position of the interface plate, ensure that there is sufficient clearance space for the locking washer on the nipple pipe. After making the bottom cutout, all external surfaces of the box are painted white. Affix the interface plate and conductive rubber gasket to the box with pan head cross screws, #8-32 5/16\", containing a rubber O-ring. Interface Plate Assembly Grind down the back cover of the weatherproof box so that it lays flat against the interface plate. Drill the center hole with the step drill and the clearance holes for the 8-32 screws. Alignment is not critical, as long as the center feedthrough connectors are within the center hole, and the distance between the outer tapped holes matches with their distance on the interface plate. Weatherbox Modifications Note the location of the green ground pin when the weather box is affixed to the interface plate. Use the provided plugs to close both side holes. Punch out the center hole and corresponding screw holes from the provided foam weatherbox gasket. Be sure to thread the extension cord wire through the lid and gasket prior to soldering the wires to the feedthroughs. Close both weatherbox lid side holes with the provided plugs. Weatherbox Assembly The 1in diameter pipe is secured on both sides by a locking washer (not pictured). The three SMAs are secured with one nut outside, one locking washer and two nuts inside. Trim the foam gasket flush to the weatherbox, and the conductive gasket flush to the interface plate. Using RTV108 translucent adhesive, seal the following edges: between the weatherbox lid and body, around all plugs in the weatherbox, around the outside of the interface plate, and around the outer locking washer of the 1in nipple pipe. Enclosure Plate The steel plate that comes provided with the enclosure box is discarded and replaced instead with an aluminum plate. Use a step drill for the four corner post holes. Drill the chassis holes on the bottom layer of the FEM assembly so that they fit 4-40 hardware. For the SNAP and GPS boards which are attached with standoffs, the screw length to affix the board to the standoff is 3/16\" and to affix the standoffs to the plate the length is 1/4\". Enclosure Plate Drawing After affixing the FEM, screw the MiniCircuits amplifiers to the SMA connectors to determine how high they sit off the plate. Fill the gap with 2-56 size washers (should be around 3). Drill the 2-56 clearance holes for the amplifiers with them in place. Choose appropriate screw length, will vary if many more or less than 3 washers used. Enclosure Lid Enclosure Lid Drawing For the RPi and PSU, the standoffs are 4-40x1/2\" hex. Screws through the boards are 3/16\" and screws through the lid are 1/4\". The switch is secured by 5/16\" screws and is backed with a nut. The LRS-50 uses metric hardware 6mm in length. Fan The fan can be attached to the lid with 1/4\" L brackets and 4-40 hardware. 5/16\" screws from the fan into the threaded side of the L bracket. 3/4\" hardware with locking washer + nut combo from L bracket through the lid. In the case where the SNAP FPGA fan is missing, the large fan is mounted directly to the SNAP board, using the three unused mounting holes on the right hand side. The .stl file for the adjustable height 3D printed bracket can be found here . Use #6 hardware to attach the fan to the base, and #2 hardware to attach the base to the SNAP. The fan is wired to the Fan terminal of the PSU using the braided red and white wire. Wiring Wiring Diagram Wiring Key * Valon * Red/white braided wire from +6Vdc to 'Valon' terminal of PSU * 086-3SMR+ cable from Source 1 SMA to top middle FEM * 086-8SMR+ cable from Ext Ref SMA to top left GPS * 086-12SMRSM+ cable from Source 2 SMA to 4th from right SNAP * FEM * 086-24SMRSM+ cable from top left to POL A on interface * 086-24SMRSM+ cable from top right to POL B on interface * 4 port terminal (splice to red/white braided wire) to FEM terminal of PSU * 4 port terminal (splice to blue/green wire) to TXD, RXD terminal of RPi * 086-4 or 086-3SM+ cable from bottom left amp to 6th from left SNAP * Red/white braided wire from bottom left amp to RPi terminal PSU * 086-4 or 086-3SM+ cable from bottom right amp to 9th from left SNAP * Red/white braided wire from bottom right amp to RPi terminal PSU * GPS * 086-15SMRSM+ cable from top right to 2nd from right SNAP * 086-15SMRSM+ cable from bottom SMA to GPS on interface * Power plug to Fan terminal of PSU * SNAP * 18 gauge red/black pair from 6 pin power plug to SNAP terminal of PSU * P1, P2 to SFP+ 1,2 on Switch * Switch * Power plug to Switch terminal of PSU * Ethernet cable from PoE In to bottom left ethernet plug of RPi * LRS-50 * 18 gauge red/black pair from V-, V+ to GND, 12V terminal of PSU * Black, white wire (16 gauge bundle) from L, N to H, N feedthroughs on interface * Green cable (16 gauge bundle) from GND to bottom ground pin * 16 gauge green cable from lid ground pin to bottom ground pin * RPi * Red/white braided wire from 5V, GND terminal to RPi terminal of PSU * Red/white braided wire from IO20, GND terminal to SW, GND of PSU Mounting Brackets Cut the square tube to 15\" plus four 1\" spacers. Using the 'X' size drill bit, drill out the 10th hole and drill a hole between the 6th and 7th holes from the uncut end. Insert the v-bolt and saddle lock into the drilled holes, fix with washer and nut. Insert bolts through the third hole in from each end. Add 1\" spacer to each, and thread through the tabs on the back of the box. Affix with a washer and nut. Mounting Brackets Final Steps Apply lock-tite to all hardware except those with nylocks. Run fiber cable through the 1 in nipple pipe into the Switch. Plug pipe with steel wool. Screw feed to the top of the box using 8/32 hex screws and nuts. The LNAs connect to the terminals on the feed, and SMA cables connect the LNA outputs to POLA and POLB on the interface plate. The magnetic GPS unit attaches to a side arm, and screws into the GPS port on the interface plate. TODO: side arm? The GReX box is designed to mount on a vertical pipe. Use the V bolt and saddle lock to adjust and tighten.","title":"The Box"},{"location":"hardware/box/#the-box","text":"Here one can find documentation re: the GReX box, including cables, routing, etc. A 3D PDF is available for download here . The BOM is available here","title":"The Box"},{"location":"hardware/box/#interface-plate","text":"The custom interface plate is machined from stock aluminum measuring approximately 12x3x1/8 inches. The drilled hole specifications are shown in the drawing below: Interface Plate Drawing Tips for machining The drilled holes are measured such that the stock size does not need to be precise. This plate will later be used to mark the corresponding holes on the main box, so the outer eight holes could be measured from all four sides. We assume the long edges arrive machined parallel. Note that the eight tapped holes around the perimeter of the piece should first be drilled with a #29 bit. After aligning the plate and marking the bottom of the box, the holes are re-drilled using the specified #18 bit. To adhere the interface plate to the main box, first the cutout must be made with a jig saw. Refer to the drawing below for placement. Note that the \"bottom\" of the box is defined when the lid is facing upwards and the hinge is on the left side. Double check that the internal ground pin is not removed or damaged with the cutout. Box Cutout Drawing Tips for machining After marking the box according to the above drawing, we marked four points: one inside each corner of the cutout, offset 3/8\" from the edge. We drilled 3/4\" holes at each point, which served as entry points for the jigsaw blade (20 TPI, no oscillation). File generously to remove sharp edges. After the cutout is finished, use the interface plate to mark the location of the eight screw holes. When determining the position of the interface plate, ensure that there is sufficient clearance space for the locking washer on the nipple pipe. After making the bottom cutout, all external surfaces of the box are painted white. Affix the interface plate and conductive rubber gasket to the box with pan head cross screws, #8-32 5/16\", containing a rubber O-ring. Interface Plate Assembly Grind down the back cover of the weatherproof box so that it lays flat against the interface plate. Drill the center hole with the step drill and the clearance holes for the 8-32 screws. Alignment is not critical, as long as the center feedthrough connectors are within the center hole, and the distance between the outer tapped holes matches with their distance on the interface plate. Weatherbox Modifications Note the location of the green ground pin when the weather box is affixed to the interface plate. Use the provided plugs to close both side holes. Punch out the center hole and corresponding screw holes from the provided foam weatherbox gasket. Be sure to thread the extension cord wire through the lid and gasket prior to soldering the wires to the feedthroughs. Close both weatherbox lid side holes with the provided plugs. Weatherbox Assembly The 1in diameter pipe is secured on both sides by a locking washer (not pictured). The three SMAs are secured with one nut outside, one locking washer and two nuts inside. Trim the foam gasket flush to the weatherbox, and the conductive gasket flush to the interface plate. Using RTV108 translucent adhesive, seal the following edges: between the weatherbox lid and body, around all plugs in the weatherbox, around the outside of the interface plate, and around the outer locking washer of the 1in nipple pipe.","title":"Interface Plate"},{"location":"hardware/box/#enclosure-plate","text":"The steel plate that comes provided with the enclosure box is discarded and replaced instead with an aluminum plate. Use a step drill for the four corner post holes. Drill the chassis holes on the bottom layer of the FEM assembly so that they fit 4-40 hardware. For the SNAP and GPS boards which are attached with standoffs, the screw length to affix the board to the standoff is 3/16\" and to affix the standoffs to the plate the length is 1/4\". Enclosure Plate Drawing After affixing the FEM, screw the MiniCircuits amplifiers to the SMA connectors to determine how high they sit off the plate. Fill the gap with 2-56 size washers (should be around 3). Drill the 2-56 clearance holes for the amplifiers with them in place. Choose appropriate screw length, will vary if many more or less than 3 washers used.","title":"Enclosure Plate"},{"location":"hardware/box/#enclosure-lid","text":"Enclosure Lid Drawing For the RPi and PSU, the standoffs are 4-40x1/2\" hex. Screws through the boards are 3/16\" and screws through the lid are 1/4\". The switch is secured by 5/16\" screws and is backed with a nut. The LRS-50 uses metric hardware 6mm in length.","title":"Enclosure Lid"},{"location":"hardware/box/#fan","text":"The fan can be attached to the lid with 1/4\" L brackets and 4-40 hardware. 5/16\" screws from the fan into the threaded side of the L bracket. 3/4\" hardware with locking washer + nut combo from L bracket through the lid. In the case where the SNAP FPGA fan is missing, the large fan is mounted directly to the SNAP board, using the three unused mounting holes on the right hand side. The .stl file for the adjustable height 3D printed bracket can be found here . Use #6 hardware to attach the fan to the base, and #2 hardware to attach the base to the SNAP. The fan is wired to the Fan terminal of the PSU using the braided red and white wire.","title":"Fan"},{"location":"hardware/box/#wiring","text":"Wiring Diagram Wiring Key * Valon * Red/white braided wire from +6Vdc to 'Valon' terminal of PSU * 086-3SMR+ cable from Source 1 SMA to top middle FEM * 086-8SMR+ cable from Ext Ref SMA to top left GPS * 086-12SMRSM+ cable from Source 2 SMA to 4th from right SNAP * FEM * 086-24SMRSM+ cable from top left to POL A on interface * 086-24SMRSM+ cable from top right to POL B on interface * 4 port terminal (splice to red/white braided wire) to FEM terminal of PSU * 4 port terminal (splice to blue/green wire) to TXD, RXD terminal of RPi * 086-4 or 086-3SM+ cable from bottom left amp to 6th from left SNAP * Red/white braided wire from bottom left amp to RPi terminal PSU * 086-4 or 086-3SM+ cable from bottom right amp to 9th from left SNAP * Red/white braided wire from bottom right amp to RPi terminal PSU * GPS * 086-15SMRSM+ cable from top right to 2nd from right SNAP * 086-15SMRSM+ cable from bottom SMA to GPS on interface * Power plug to Fan terminal of PSU * SNAP * 18 gauge red/black pair from 6 pin power plug to SNAP terminal of PSU * P1, P2 to SFP+ 1,2 on Switch * Switch * Power plug to Switch terminal of PSU * Ethernet cable from PoE In to bottom left ethernet plug of RPi * LRS-50 * 18 gauge red/black pair from V-, V+ to GND, 12V terminal of PSU * Black, white wire (16 gauge bundle) from L, N to H, N feedthroughs on interface * Green cable (16 gauge bundle) from GND to bottom ground pin * 16 gauge green cable from lid ground pin to bottom ground pin * RPi * Red/white braided wire from 5V, GND terminal to RPi terminal of PSU * Red/white braided wire from IO20, GND terminal to SW, GND of PSU","title":"Wiring"},{"location":"hardware/box/#mounting-brackets","text":"Cut the square tube to 15\" plus four 1\" spacers. Using the 'X' size drill bit, drill out the 10th hole and drill a hole between the 6th and 7th holes from the uncut end. Insert the v-bolt and saddle lock into the drilled holes, fix with washer and nut. Insert bolts through the third hole in from each end. Add 1\" spacer to each, and thread through the tabs on the back of the box. Affix with a washer and nut. Mounting Brackets","title":"Mounting Brackets"},{"location":"hardware/box/#final-steps","text":"Apply lock-tite to all hardware except those with nylocks. Run fiber cable through the 1 in nipple pipe into the Switch. Plug pipe with steel wool. Screw feed to the top of the box using 8/32 hex screws and nuts. The LNAs connect to the terminals on the feed, and SMA cables connect the LNA outputs to POLA and POLB on the interface plate. The magnetic GPS unit attaches to a side arm, and screws into the GPS port on the interface plate. TODO: side arm? The GReX box is designed to mount on a vertical pipe. Use the V bolt and saddle lock to adjust and tighten.","title":"Final Steps"},{"location":"hardware/feed/","text":"Feed Antenna TODO! Jonas info","title":"Feed Antenna"},{"location":"hardware/feed/#feed-antenna","text":"TODO! Jonas info","title":"Feed Antenna"},{"location":"hardware/fem/","text":"Frontend Module The frontend module (FEM) is a device that performs the analog signal processing after the LNAs. This includes filtering, downconversion, and amplification. Additionally, this module provides rudimentary monitor and control support. Bare PCB Completed Module Hardware Design The hardware design itself is implemented in the free KiCAD program and is available here . The current hardware uses ENIG to help reflow of the fine-pitch components. Additionally, it utilizes the low-loss RO4003C substrate in a custom 4-layer stackup. Schematics BOM Case Here is an annotated view of the PCB with major system components labeled: Software Design The RF hardware mostly operates without the intervention of any software. The only step required to use the RF hardware is to set the valid attenuation level, which defaults to 0 dB. As such, the primary goal of the digital section of the FEM is to perform Monitor and Control (MnC). MnC is achieved via an 115200 baud 3.3V UART interface on the main connector. The firmware design is carried out in the Rust programming language, and whose source can be found here . In that repository, one can also find the control software, which is used to perform the monitoring and controlling. With this software, there is control for enabling/disabling the LNA bias and setting the interstage IF attenuator. For the digital attenuator, we can set attenuation (and therefore change the total gain) from 0 dB to 31.5 dB. This attenuator is used to maximize the dynamic range of the ADC and can be set for environmental RFI levels. CLI To use the CLI program, connect the FEM to a serial port either with a USB to serial adapter or directly to a hardware UART port (like on the Raspberry Pi). If the pi doesn't have the cli program already copied over, it can be found here . Usage: cli Commands: mon Gets monitor data from the FEM lna Controls the power of the LNA if Sets the IF \"power good\" threshold atten Sets the attenuation level in dB ( 0 to 31 .5 ) help Print this message or the help of the given subcommand ( s ) Arguments: Serial port for the FEM Options: -h, --help Print help -V, --version Print version To set the power state of the LNAs, use: ./cli lna ch< 1 | 2 > To set the attenuation: ./cli atten < 0 -31.5> To grab the monitor data: ./cli mon Physical Interface There are eight LEDs on the front panel. Four red LEDs to indicate power statess, two blue LEDs for serial activity, and two green LEDs for system status. The green LEDs will be enabled when the IF power is at a nominal level.","title":"Fronend Module"},{"location":"hardware/fem/#frontend-module","text":"The frontend module (FEM) is a device that performs the analog signal processing after the LNAs. This includes filtering, downconversion, and amplification. Additionally, this module provides rudimentary monitor and control support. Bare PCB Completed Module","title":"Frontend Module"},{"location":"hardware/fem/#hardware-design","text":"The hardware design itself is implemented in the free KiCAD program and is available here . The current hardware uses ENIG to help reflow of the fine-pitch components. Additionally, it utilizes the low-loss RO4003C substrate in a custom 4-layer stackup. Schematics BOM Case Here is an annotated view of the PCB with major system components labeled:","title":"Hardware Design"},{"location":"hardware/fem/#software-design","text":"The RF hardware mostly operates without the intervention of any software. The only step required to use the RF hardware is to set the valid attenuation level, which defaults to 0 dB. As such, the primary goal of the digital section of the FEM is to perform Monitor and Control (MnC). MnC is achieved via an 115200 baud 3.3V UART interface on the main connector. The firmware design is carried out in the Rust programming language, and whose source can be found here . In that repository, one can also find the control software, which is used to perform the monitoring and controlling. With this software, there is control for enabling/disabling the LNA bias and setting the interstage IF attenuator. For the digital attenuator, we can set attenuation (and therefore change the total gain) from 0 dB to 31.5 dB. This attenuator is used to maximize the dynamic range of the ADC and can be set for environmental RFI levels.","title":"Software Design"},{"location":"hardware/fem/#cli","text":"To use the CLI program, connect the FEM to a serial port either with a USB to serial adapter or directly to a hardware UART port (like on the Raspberry Pi). If the pi doesn't have the cli program already copied over, it can be found here . Usage: cli Commands: mon Gets monitor data from the FEM lna Controls the power of the LNA if Sets the IF \"power good\" threshold atten Sets the attenuation level in dB ( 0 to 31 .5 ) help Print this message or the help of the given subcommand ( s ) Arguments: Serial port for the FEM Options: -h, --help Print help -V, --version Print version To set the power state of the LNAs, use: ./cli lna ch< 1 | 2 > To set the attenuation: ./cli atten < 0 -31.5> To grab the monitor data: ./cli mon","title":"CLI"},{"location":"hardware/fem/#physical-interface","text":"There are eight LEDs on the front panel. Four red LEDs to indicate power statess, two blue LEDs for serial activity, and two green LEDs for system status. The green LEDs will be enabled when the IF power is at a nominal level.","title":"Physical Interface"},{"location":"hardware/fpga/","text":"Digital Backend TODO! Describe hardware interfaces of SNAP.","title":"Digital Backend"},{"location":"hardware/fpga/#digital-backend","text":"TODO! Describe hardware interfaces of SNAP.","title":"Digital Backend"},{"location":"hardware/overview/","text":"Hardware Overview The GReX hardware system has several \"top level\" components, which constitute the entire system. These include the feed antenna and low noise amplifiers (LNA), the frontend module , the digital backend , and of course the server. The following diagrams lay out general overview of the interconnections. Showing them all at once would be a bit much, so they're broken down here into discrete kinds of signals. RF Signal Path flowchart TD A[Feed] B[FEM] C[SNAP] D[Server] A --> L1[LNA] A --> L2[LNA] L1 -->|H Pol| B L2 -->|V Pol| B subgraph The Box B -->|H Pol| C B -->|V Pol| C end C -->|10 GbE| D[Server] Power Distribution flowchart BT L1[LNA] L2[LNA] B[FEM] C[SNAP] S[Switching Supply] R[Custom Linear PSU] P[Raspberry Pi] M[Mains Power] V[Synthesizer] G[GPS Receiver] M -->|120-240V AC| S subgraph The Box R -->|6.5V DC| V R -->|12V DC| G S -->|12V DC| C S -->|12V DC| R R -->|6.5V DC| B R -->|5V DC| P end B --->|5.5V DC| L1 B --->|5.5V DC| L2 Clocks, References and Timing flowchart BT B[FEM] V[Synthesizer] G[GPS Receiver] S[SNAP] A[GPS Antenna] subgraph The Box G -->|10 MHz| V G -->|PPS| S V -->|500 MHz| S V -->|1030 MHz| B end A --> G Monitor and Control flowchart TB B[FEM] S[SNAP] P[Raspberry Pi] D[Server] subgraph The Box P <-->|UART| B S <-->|GPIO| P end P <--->|1 GbE| D","title":"Overview"},{"location":"hardware/overview/#hardware-overview","text":"The GReX hardware system has several \"top level\" components, which constitute the entire system. These include the feed antenna and low noise amplifiers (LNA), the frontend module , the digital backend , and of course the server. The following diagrams lay out general overview of the interconnections. Showing them all at once would be a bit much, so they're broken down here into discrete kinds of signals.","title":"Hardware Overview"},{"location":"hardware/overview/#rf-signal-path","text":"flowchart TD A[Feed] B[FEM] C[SNAP] D[Server] A --> L1[LNA] A --> L2[LNA] L1 -->|H Pol| B L2 -->|V Pol| B subgraph The Box B -->|H Pol| C B -->|V Pol| C end C -->|10 GbE| D[Server]","title":"RF Signal Path"},{"location":"hardware/overview/#power-distribution","text":"flowchart BT L1[LNA] L2[LNA] B[FEM] C[SNAP] S[Switching Supply] R[Custom Linear PSU] P[Raspberry Pi] M[Mains Power] V[Synthesizer] G[GPS Receiver] M -->|120-240V AC| S subgraph The Box R -->|6.5V DC| V R -->|12V DC| G S -->|12V DC| C S -->|12V DC| R R -->|6.5V DC| B R -->|5V DC| P end B --->|5.5V DC| L1 B --->|5.5V DC| L2","title":"Power Distribution"},{"location":"hardware/overview/#clocks-references-and-timing","text":"flowchart BT B[FEM] V[Synthesizer] G[GPS Receiver] S[SNAP] A[GPS Antenna] subgraph The Box G -->|10 MHz| V G -->|PPS| S V -->|500 MHz| S V -->|1030 MHz| B end A --> G","title":"Clocks, References and Timing"},{"location":"hardware/overview/#monitor-and-control","text":"flowchart TB B[FEM] S[SNAP] P[Raspberry Pi] D[Server] subgraph The Box P <-->|UART| B S <-->|GPIO| P end P <--->|1 GbE| D","title":"Monitor and Control"},{"location":"software/box/","text":"Box Software There are four devices that require some software configuration: the Valon synthesizer, the SNAP FPGA, the 10 GbE switch, and the Raspberry Pi. These configuration steps should already be performed before we ship a box, but for completeness, here are the steps that we performed. Valon We need to configure the valon synthesizer to act as the LO for the downconverter and the reference clock for the SNAP ADC. Use the GUI tool here to load this configuration file. Next, go to synthesizer -> write registers. Then, save the configuration to flash to preserve this configuration across reboots. Switch With the box connected and powered on, create an SSH relay to the switch's configuration interface with ssh -L 8291 :192.168.88.1:8291 user@ Then, using winbox connect to localhost, select files on the left, and upload this config file . This should trigger a reboot. Raspberry Pi We prepared the RPi image using the standard raspbian lite OS . As part of the initial image creation, we set the hostname to grex-pi and enabled password-based SSH. Using raspi-config , we did the following: - disabled the serial login shell - enabled the hardware serial interface Then, we disabled the hardware's radios by modifying the config.txt file like so . Then, we configured the Pi to have the static IP address of 192.168.0.2 by following this Then, we disabled HCI UART by running sudo systemctl disable hciuart SNAP See SNAP Setup","title":"Box Setup"},{"location":"software/box/#box-software","text":"There are four devices that require some software configuration: the Valon synthesizer, the SNAP FPGA, the 10 GbE switch, and the Raspberry Pi. These configuration steps should already be performed before we ship a box, but for completeness, here are the steps that we performed.","title":"Box Software"},{"location":"software/box/#valon","text":"We need to configure the valon synthesizer to act as the LO for the downconverter and the reference clock for the SNAP ADC. Use the GUI tool here to load this configuration file. Next, go to synthesizer -> write registers. Then, save the configuration to flash to preserve this configuration across reboots.","title":"Valon"},{"location":"software/box/#switch","text":"With the box connected and powered on, create an SSH relay to the switch's configuration interface with ssh -L 8291 :192.168.88.1:8291 user@ Then, using winbox connect to localhost, select files on the left, and upload this config file . This should trigger a reboot.","title":"Switch"},{"location":"software/box/#raspberry-pi","text":"We prepared the RPi image using the standard raspbian lite OS . As part of the initial image creation, we set the hostname to grex-pi and enabled password-based SSH. Using raspi-config , we did the following: - disabled the serial login shell - enabled the hardware serial interface Then, we disabled the hardware's radios by modifying the config.txt file like so . Then, we configured the Pi to have the static IP address of 192.168.0.2 by following this Then, we disabled HCI UART by running sudo systemctl disable hciuart","title":"Raspberry Pi"},{"location":"software/box/#snap","text":"See SNAP Setup","title":"SNAP"},{"location":"software/data_processing/","text":"Data Processing Voltage Dumps The voltage dumps are stored in NetCDF , a machine-independent data format intended for array-oriented scientific data such as ours. This format is a subset of HDF5 and is supported by all major programming languages. Not only that, but it is self describing! Our files contain axis variables on each dimension to remove any ambiguity as to which coordinate each data point refers to. Using the NetCDF binaries, we can quickly introspect a voltage dump to see how it is laid out user@grex:/hdd/data/voltages$ ncdump -h grex_dump-20240319T182522.nc netcdf grex_dump-20240319T182522 { dimensions: time = 1048576 ; pol = 2 ; freq = 2048 ; reim = 2 ; variables: double time ( time ) ; time:units = \"Days\" ; time:long_name = \"TAI days since the MJD Epoch\" ; string pol ( pol ) ; pol:long_name = \"Polarization\" ; double freq ( freq ) ; freq:units = \"Megahertz\" ; freq:long_name = \"Frequency\" ; string reim ( reim ) ; reim:long_name = \"Complex\" ; byte voltages ( time, pol, freq, reim ) ; voltages:long_name = \"Channelized Voltages\" ; voltages:units = \"Volts\" ; } Part of the trickiness here is HDF5 (and NetCDF by extension) doesn't support complex numbers, so the real and imaginary components are stored independently as another dimension. Python Reading Example In Python, an excellent library for dealing with dimensional data is xarray . This library supports a large number of file formats, which are optional dependencies. For us, we need the netCDF4 python package, but it's probably best just to use the complete installation. It might also help if you read the xarray page on NetCDF . Reading this file with xarray . open_dataset ( \"grex_dump.nc\" ) will associate all the axes to the appropriate dimensions, we just have to construct the complex numbers manually. A one-liner to do this would be voltages = ds [ \"voltages\" ] . sel ( reim = \"real\" ) + ds [ \"voltages\" ] . sel ( reim = \"imaginary\" ) * 1 j This may take a while for large dumps though, and you may want to slice it up into chunks (not sure if there is an elegant way to do that here). But now you can do eveything xarray can do! A few examples: Stokes I Calculated by taking the time-average of the sum of magnitude squared of the voltages stokesi = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = 'pol' , dtype = 'int32' ) . sum ( dim = 'reim' , dtype = 'int32' ) Note that the original voltages are in \"int8\" format which is not enough to store squared voltages, and thus we are converting to \"int32\". H1 Line Get the Stokes intensity around the H1 line (+/- 1 MHz) as a function a time h1 = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = \"pol\" , dtype = 'int32' ) . sel ( freq = slice ( 1421 , 1419 )) . sum ( dim = \"freq\" , dtype = 'int32' )","title":"Data Processing"},{"location":"software/data_processing/#data-processing","text":"","title":"Data Processing"},{"location":"software/data_processing/#voltage-dumps","text":"The voltage dumps are stored in NetCDF , a machine-independent data format intended for array-oriented scientific data such as ours. This format is a subset of HDF5 and is supported by all major programming languages. Not only that, but it is self describing! Our files contain axis variables on each dimension to remove any ambiguity as to which coordinate each data point refers to. Using the NetCDF binaries, we can quickly introspect a voltage dump to see how it is laid out user@grex:/hdd/data/voltages$ ncdump -h grex_dump-20240319T182522.nc netcdf grex_dump-20240319T182522 { dimensions: time = 1048576 ; pol = 2 ; freq = 2048 ; reim = 2 ; variables: double time ( time ) ; time:units = \"Days\" ; time:long_name = \"TAI days since the MJD Epoch\" ; string pol ( pol ) ; pol:long_name = \"Polarization\" ; double freq ( freq ) ; freq:units = \"Megahertz\" ; freq:long_name = \"Frequency\" ; string reim ( reim ) ; reim:long_name = \"Complex\" ; byte voltages ( time, pol, freq, reim ) ; voltages:long_name = \"Channelized Voltages\" ; voltages:units = \"Volts\" ; } Part of the trickiness here is HDF5 (and NetCDF by extension) doesn't support complex numbers, so the real and imaginary components are stored independently as another dimension.","title":"Voltage Dumps"},{"location":"software/data_processing/#python-reading-example","text":"In Python, an excellent library for dealing with dimensional data is xarray . This library supports a large number of file formats, which are optional dependencies. For us, we need the netCDF4 python package, but it's probably best just to use the complete installation. It might also help if you read the xarray page on NetCDF . Reading this file with xarray . open_dataset ( \"grex_dump.nc\" ) will associate all the axes to the appropriate dimensions, we just have to construct the complex numbers manually. A one-liner to do this would be voltages = ds [ \"voltages\" ] . sel ( reim = \"real\" ) + ds [ \"voltages\" ] . sel ( reim = \"imaginary\" ) * 1 j This may take a while for large dumps though, and you may want to slice it up into chunks (not sure if there is an elegant way to do that here). But now you can do eveything xarray can do! A few examples:","title":"Python Reading Example"},{"location":"software/data_processing/#stokes-i","text":"Calculated by taking the time-average of the sum of magnitude squared of the voltages stokesi = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = 'pol' , dtype = 'int32' ) . sum ( dim = 'reim' , dtype = 'int32' ) Note that the original voltages are in \"int8\" format which is not enough to store squared voltages, and thus we are converting to \"int32\".","title":"Stokes I"},{"location":"software/data_processing/#h1-line","text":"Get the Stokes intensity around the H1 line (+/- 1 MHz) as a function a time h1 = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = \"pol\" , dtype = 'int32' ) . sel ( freq = slice ( 1421 , 1419 )) . sum ( dim = \"freq\" , dtype = 'int32' )","title":"H1 Line"},{"location":"software/operation/","text":"Operation We'll flesh out all the details on operation soon, but in the meantime here are the critical details. Turning on the SNAP To turn on the SNAP, SSH into the Pi (as discussed in the server setup), Then on the pi create (if it doesn't already exist) a bash script called snap.sh with the following: Warning This is assuming you have a V2 power supply, ON and OFF are reversed otherwise #!/bin/env bash # Usage: ./snap.sh BASE_GPIO_PATH = /sys/class/gpio PWN_PIN = 20 if [ ! -e $BASE_GPIO_PATH /gpio $PWN_PIN ] ; then echo \"20\" > $BASE_GPIO_PATH /export fi echo \"out\" > $BASE_GPIO_PATH /gpio $PWN_PIN /direction if [[ -z $1 ]] ; then echo \"Please pass `on` or `off` as an argument\" else case $1 in \"on\" | \"ON\" ) echo \"0\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; \"off\" | \"OFF\" ) echo \"1\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; * ) echo \"Please pass `on` or `off` as an argument\" exit -1 ;; esac fi exit 0 Make it executable is chmod +x snap.sh , and use ./snap.sh to control the power state of the SNAP. If you get permission errors when doing so, make sure your Pi's user is a member of the gpio group. You can add your user to the group with: sudo usermod -a -G gpio $( whoami ) then relog and try again. Running the Pipeline In the grex folder, under pipeline there is the single bash script that runs the pipeline. Simply calling it ./grex.sh should start everything up. By default, it will run the normal detection pipeline. If you want to just run T0 (packet capture and first stage processing), remove the final line that calls psrdada and replace it (the whole line) with filterbank . Triggering voltage dumps Normally, T2 will send triggers to T0 to dump the voltage ringbuffer to disk. You can emulate this by sending a UDP packet to the trigger socket any other way. One simple way is with bash echo \" \" > /dev/udp/localhost/65432 SSH Port Tunneling In some circumstances, it may be useful to access ports on the GReX server on your local computer remotely. We can accomplish this using SSH Tunneling . One example of why we might want to do this is to access the 10 GbE switch configuration that is located in the far-side GReX box. It runs a normal web page on a static ip of 192.168.88.1 . You can access this from a web browser if you are sitting at the GReX server, but not remotely. To access it using SSH tunneling, we can forward that IP's port 80 (standard HTTP) to our local computer at some unused, non-privaleged port. ssh -L 8080 :192.168.88.1:80 username@grex-server-address Another example is perhaps you want to run a Jupypter Hub instance on the GReX server. In that case, the website it is hosting is on the server itself, so you would run: ssh -L 8080 :localhost:80 username@grex-server-address Another useful one is access to the Prometheus time-series database used for monitoring. That is active on port 9090 ssh -L 9090 :localhost:9090 username@grex-server-address Pulse Injection If you need to generate and inject fake pulses into raw voltages to test the pipeline, Liam Connor's injection codes contain all the relevant tools. In a Python Jupyter notebook, you can import this Python script and use the functions within to generate a fake pulse and write it to an output .dat file. import simulate_frb dt = 8.192e-6 width_sec = 2 * dt # depending on your preferred pulse width Nfreq = 2048 data , params = simulate_frb . gen_simulated_frb ( NFREQ = Nfreq , NTIME = 16384 , sim = True , fluence = 70 , # This controls the pulse SNR spec_ind = 0.0 , width = width_sec , dm = 100. , # dispersion measure background_noise = np . zeros ([ 2048 , 16384 ]), delta_t = dt , plot_burst = False , freq = ( 1530.0 , 1280.0 ), FREQ_REF = 1405.0 , scintillate = False , scat_tau_ref = 0.0 , disp_ind = 2.0 , conv_dmsmear = False , ) Before writing to an output file, you might want to check what the pulse looks like and its histogram distribution, ensuring it\u2019s not all zeros due to conversion to int8 (bit type of raw voltages). from matplotlib import pyplot as plt plt . hist ( data . astype ( np . int8 ) . flatten (), bins = 50 ) plt . semilogy () plt . show () ### also, visualize the pulse itself plt . imshow ( data . astype ( np . int8 ), aspect = 'auto' ) plt . colorbar () plt . show () If the pulse looks reasonable, we can convert to int8 type and write to an output .dat file. with open ( 'test_injpulse0.dat' , 'wb' ) as f : f . write ( data . astype ( np . int8 ) . tobytes ()) Move this .dat file to ~/grex/pipeline/fake/ , the specified directory in grex.sh , to actually inject the pulse. Then you could change the injection cadence (in seconds) in ~/grex/pipeline/.env by adding the following line. injection_cadence = 300 Now you are good to go. Querying Historical Data As part of the server setup, we've hooked up a time-series database (Prometheus) to T0, which is used to monitor the state of the telescope. This includes recording timestamped values for temperature, ADC counts, and integrated Stokes I data. It may be useful to query this data, which is relativly easy to do programatically using Prometheus' HTTP API . You can explore the database by accessing http://localhost:9090 on the server (or with SSH-forwarding described above). H1 Example Say you want to query historical Stokes I data around the H1 frequency of 1420 MHz. This data is stored in \"channels\" where the index is the FPGA's channel number. If you recall, we operate in the first Nyquist zone, so the spectrum is flipped, where channel 0 represents 1530 MHz and channel 2048 represents 1280 MHz. Using the requests library in Python, we can perform the HTTP request to the database. import requests import time import numpy as npc # Get the current (UNIX) time now = time . time () # We want data, say two weeks back past = now - 60 * 60 * 24 * 7 * 2 # We want data around 1420 MHz, so around channel 893 # Might as well grab the block of data around it, so we can watch it change in frequency with time channels = [ 889 , 890 , 891 , 892 , 893 , 894 , 895 ] data = [] uri_base = 'http://127.0.0.1:9090/api/v1/query_range' for channel in channels : params = { \"query\" : f 'spectrum {{ channel=\" { channel } \" }} ' , \"start\" : then , \"end\" : now , \"step\" : \"10m\" } # Depending on the timespan, there's a maximum size per query resp = requests . get ( uri_base , params = params ) # Extract the data and convert to floats vals = resp . json ()[ 'data' ][ 'result' ][ 0 ][ 'values' ] data . append ( np . array ( vals ) . astype ( float )) # Restructure the data into a tensor data = np . stack ( data ) # The time axis is duplicated many times, we can extract it by looking at one chunk timestamps = data [ 0 ,:, 0 ] # And then finally extract our 2D block of time/freq linear power data # Indexed in [channel, time] spectra = data [:,:, 1 ]","title":"Operation"},{"location":"software/operation/#operation","text":"We'll flesh out all the details on operation soon, but in the meantime here are the critical details.","title":"Operation"},{"location":"software/operation/#turning-on-the-snap","text":"To turn on the SNAP, SSH into the Pi (as discussed in the server setup), Then on the pi create (if it doesn't already exist) a bash script called snap.sh with the following: Warning This is assuming you have a V2 power supply, ON and OFF are reversed otherwise #!/bin/env bash # Usage: ./snap.sh BASE_GPIO_PATH = /sys/class/gpio PWN_PIN = 20 if [ ! -e $BASE_GPIO_PATH /gpio $PWN_PIN ] ; then echo \"20\" > $BASE_GPIO_PATH /export fi echo \"out\" > $BASE_GPIO_PATH /gpio $PWN_PIN /direction if [[ -z $1 ]] ; then echo \"Please pass `on` or `off` as an argument\" else case $1 in \"on\" | \"ON\" ) echo \"0\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; \"off\" | \"OFF\" ) echo \"1\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; * ) echo \"Please pass `on` or `off` as an argument\" exit -1 ;; esac fi exit 0 Make it executable is chmod +x snap.sh , and use ./snap.sh to control the power state of the SNAP. If you get permission errors when doing so, make sure your Pi's user is a member of the gpio group. You can add your user to the group with: sudo usermod -a -G gpio $( whoami ) then relog and try again.","title":"Turning on the SNAP"},{"location":"software/operation/#running-the-pipeline","text":"In the grex folder, under pipeline there is the single bash script that runs the pipeline. Simply calling it ./grex.sh should start everything up. By default, it will run the normal detection pipeline. If you want to just run T0 (packet capture and first stage processing), remove the final line that calls psrdada and replace it (the whole line) with filterbank .","title":"Running the Pipeline"},{"location":"software/operation/#triggering-voltage-dumps","text":"Normally, T2 will send triggers to T0 to dump the voltage ringbuffer to disk. You can emulate this by sending a UDP packet to the trigger socket any other way. One simple way is with bash echo \" \" > /dev/udp/localhost/65432","title":"Triggering voltage dumps"},{"location":"software/operation/#ssh-port-tunneling","text":"In some circumstances, it may be useful to access ports on the GReX server on your local computer remotely. We can accomplish this using SSH Tunneling . One example of why we might want to do this is to access the 10 GbE switch configuration that is located in the far-side GReX box. It runs a normal web page on a static ip of 192.168.88.1 . You can access this from a web browser if you are sitting at the GReX server, but not remotely. To access it using SSH tunneling, we can forward that IP's port 80 (standard HTTP) to our local computer at some unused, non-privaleged port. ssh -L 8080 :192.168.88.1:80 username@grex-server-address Another example is perhaps you want to run a Jupypter Hub instance on the GReX server. In that case, the website it is hosting is on the server itself, so you would run: ssh -L 8080 :localhost:80 username@grex-server-address Another useful one is access to the Prometheus time-series database used for monitoring. That is active on port 9090 ssh -L 9090 :localhost:9090 username@grex-server-address","title":"SSH Port Tunneling"},{"location":"software/operation/#pulse-injection","text":"If you need to generate and inject fake pulses into raw voltages to test the pipeline, Liam Connor's injection codes contain all the relevant tools. In a Python Jupyter notebook, you can import this Python script and use the functions within to generate a fake pulse and write it to an output .dat file. import simulate_frb dt = 8.192e-6 width_sec = 2 * dt # depending on your preferred pulse width Nfreq = 2048 data , params = simulate_frb . gen_simulated_frb ( NFREQ = Nfreq , NTIME = 16384 , sim = True , fluence = 70 , # This controls the pulse SNR spec_ind = 0.0 , width = width_sec , dm = 100. , # dispersion measure background_noise = np . zeros ([ 2048 , 16384 ]), delta_t = dt , plot_burst = False , freq = ( 1530.0 , 1280.0 ), FREQ_REF = 1405.0 , scintillate = False , scat_tau_ref = 0.0 , disp_ind = 2.0 , conv_dmsmear = False , ) Before writing to an output file, you might want to check what the pulse looks like and its histogram distribution, ensuring it\u2019s not all zeros due to conversion to int8 (bit type of raw voltages). from matplotlib import pyplot as plt plt . hist ( data . astype ( np . int8 ) . flatten (), bins = 50 ) plt . semilogy () plt . show () ### also, visualize the pulse itself plt . imshow ( data . astype ( np . int8 ), aspect = 'auto' ) plt . colorbar () plt . show () If the pulse looks reasonable, we can convert to int8 type and write to an output .dat file. with open ( 'test_injpulse0.dat' , 'wb' ) as f : f . write ( data . astype ( np . int8 ) . tobytes ()) Move this .dat file to ~/grex/pipeline/fake/ , the specified directory in grex.sh , to actually inject the pulse. Then you could change the injection cadence (in seconds) in ~/grex/pipeline/.env by adding the following line. injection_cadence = 300 Now you are good to go.","title":"Pulse Injection"},{"location":"software/operation/#querying-historical-data","text":"As part of the server setup, we've hooked up a time-series database (Prometheus) to T0, which is used to monitor the state of the telescope. This includes recording timestamped values for temperature, ADC counts, and integrated Stokes I data. It may be useful to query this data, which is relativly easy to do programatically using Prometheus' HTTP API . You can explore the database by accessing http://localhost:9090 on the server (or with SSH-forwarding described above).","title":"Querying Historical Data"},{"location":"software/operation/#h1-example","text":"Say you want to query historical Stokes I data around the H1 frequency of 1420 MHz. This data is stored in \"channels\" where the index is the FPGA's channel number. If you recall, we operate in the first Nyquist zone, so the spectrum is flipped, where channel 0 represents 1530 MHz and channel 2048 represents 1280 MHz. Using the requests library in Python, we can perform the HTTP request to the database. import requests import time import numpy as npc # Get the current (UNIX) time now = time . time () # We want data, say two weeks back past = now - 60 * 60 * 24 * 7 * 2 # We want data around 1420 MHz, so around channel 893 # Might as well grab the block of data around it, so we can watch it change in frequency with time channels = [ 889 , 890 , 891 , 892 , 893 , 894 , 895 ] data = [] uri_base = 'http://127.0.0.1:9090/api/v1/query_range' for channel in channels : params = { \"query\" : f 'spectrum {{ channel=\" { channel } \" }} ' , \"start\" : then , \"end\" : now , \"step\" : \"10m\" } # Depending on the timespan, there's a maximum size per query resp = requests . get ( uri_base , params = params ) # Extract the data and convert to floats vals = resp . json ()[ 'data' ][ 'result' ][ 0 ][ 'values' ] data . append ( np . array ( vals ) . astype ( float )) # Restructure the data into a tensor data = np . stack ( data ) # The time axis is duplicated many times, we can extract it by looking at one chunk timestamps = data [ 0 ,:, 0 ] # And then finally extract our 2D block of time/freq linear power data # Indexed in [channel, time] spectra = data [:,:, 1 ]","title":"H1 Example"},{"location":"software/overview/","text":"Software Overview There are two primary components to the software stack in GReX. First, the SNAP board must be configured and setup to send voltage data to the server. After that, the pipeline software should take care of the rest. Pipeline Overview flowchart TD A[SNAP] -->|UDP| B[T0] B -->|PSRDADA| C[RFI Cleaning] C -->|PSRDADA| D[Heimdall] D -->|Sockets| E[T2] Software Manifesto To limit downtime and maximize reproducability, we will try to adopt a consistent software development strategy. Primarily: Builds will be deterministic and reproducible Code will be version controlled, organized, and public Language Specific Rust No clippy warnings Avoid unsafe Document everything Formatted with rustfmt C++ Code will be formatted with clang-format's LLVM style Try to minimize (solve) all errors from -Wall Python Code will be formatted with Black Docstrings will follow the numpy format Gradual typing will be used (PEP 438) and checked with mypy or equivalent Environments will have pinned dependencies (reproducible) Most of this can be accomplised by using Poetry .","title":"Overview"},{"location":"software/overview/#software-overview","text":"There are two primary components to the software stack in GReX. First, the SNAP board must be configured and setup to send voltage data to the server. After that, the pipeline software should take care of the rest.","title":"Software Overview"},{"location":"software/overview/#pipeline-overview","text":"flowchart TD A[SNAP] -->|UDP| B[T0] B -->|PSRDADA| C[RFI Cleaning] C -->|PSRDADA| D[Heimdall] D -->|Sockets| E[T2]","title":"Pipeline Overview"},{"location":"software/overview/#software-manifesto","text":"To limit downtime and maximize reproducability, we will try to adopt a consistent software development strategy. Primarily: Builds will be deterministic and reproducible Code will be version controlled, organized, and public","title":"Software Manifesto"},{"location":"software/overview/#language-specific","text":"","title":"Language Specific"},{"location":"software/overview/#rust","text":"No clippy warnings Avoid unsafe Document everything Formatted with rustfmt","title":"Rust"},{"location":"software/overview/#c","text":"Code will be formatted with clang-format's LLVM style Try to minimize (solve) all errors from -Wall","title":"C++"},{"location":"software/overview/#python","text":"Code will be formatted with Black Docstrings will follow the numpy format Gradual typing will be used (PEP 438) and checked with mypy or equivalent Environments will have pinned dependencies (reproducible) Most of this can be accomplised by using Poetry .","title":"Python"},{"location":"software/pipeline/","text":"Pipeline","title":"Pipeline"},{"location":"software/pipeline/#pipeline","text":"","title":"Pipeline"},{"location":"software/server_setup/","text":"Server Setup Once you get your server (either from Puget systems or otherwise), we need to setup additional hardware, adjust some system settings, setup networking, and install the pipeline software. Hardware Setup The server straight from Puget does not have the GPU or 10 GbE network card installed, we will do this first. Open the case and remove the GPU retention bracket Remove the test GPU (T1000), keep this for later in case we need an aditional video output Install the 3090 Ti in the first GPU slot, this will take up three slots of space Install the network card in the bottom slot Wire the 3090 Ti power cable to the harness provided by Puget (they knew this was the GPU we were going to install) Remove the GPU retention clips from the retention bracket that would interfere with the card. It's too tall for it anyway. Replace the retention bracket and close the case. Finally, we need to hook up a monitor to the 3090 Ti so we can setup the software Initial OS Setup On boot, you will be presented with the Ubuntu graphical interface. If a password is requested, the deafult is provided in the information booklet that came with the hardware. First, we will change the system password. Set this to something memorable. passwd $( whoami ) Now, we will update the system. There shouldn't be many updates if this is a new machine. sudo apt-get update sudo apt-get upgrade -y We will be editing a bunch of files, if you are comfy in the command line, you probably want to install some editors. Otherwise the graphical gedit tool will be fine. sudo apt-get install emacs vim -y Finally, we will set the hostname. We'll be using the grex-- paradigm (just for clarity, no real reason not to). As in, the first server that is managed by Caltech at OVRO will be grex-caltech-ovro . sudo hostnamectl set-hostname Some updates may require a reboot. If it asks, do that now. Networking Now, we need to setup the networking for the GReX system. We will operate under the assumption that the internet-facing connection will get an IP address from a DHCP server. If that is not the case, consult whoever runs your network on the appropriate setup. Regardless of the WAN connection, the 10 GbE fiber connection to the GReX terminal will be configured the same. Overview The 10 GbE fiber port serves a few purposes. It is the main data transfer link between the FPGA in the field and the server, but it also carries the monitor and control for the box. This monitor and control connection includes the SNAP control connection and the Raspberry Pi. The SNAP requires an external DHCP server, which we have to provide on this port. Additionally, the 10 GbE switch in the box has its own default subnet for configuration ( 192.168.88.X ). To make everything talk to each other, we need to add two IPs on this port: one in the subnet of the switch's config interface, and the other for DHCP of the various devices. Netplan In /etc/netplan remove any files that are currently there. Check whether you are using NetworkManager or networkd: systemctl status NetworkManager systemctl status systemd-networkd If NetworkManager is running and networkd is not, disable NetworkManager and enable networkd. (Otherwise, skip this step.) sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo systemctl enable systemd-networkd Then, create a new file called config.yaml with the following contents network : version : 2 renderer : networkd ethernets : # Two WAN interfaces. Configure this according to your network setup enp36s0f0 : dhcp4 : true enp36s0f1 : dhcp4 : true # 10 GbE connection over fiber to the box enp1s0f0 : mtu : 9000 addresses : - 192.168.0.1/24 - 192.168.88.2/24 Then apply with sudo netplan apply DHCP Server Now, we need to setup the DHCP server on the 10 GbE port. First, we install the DHCP server software: sudo apt-get install dnsmasq Create the configuration file in /etc/dnsmasq.conf # Only bind to the 10 GbE interface interface = enp1s0f0 # Disable DNS port = 0 # DHCP Options dhcp-range = 192.168.0.0,static dhcp-option = option:router,192.168.0.1 dhcp-option = option:netmask,255.255.255.0 #dhcp-host=,192.168.0.3,snap log-async log-queries log-dhcp Then, enable the DHCP server service sudo systemctl enable dnsmasq --now This sets up a very simple DHCP server that will give the IP address 192.168.0.3 to the SNAP. Unfortunately, the folks who set up the networking interface for the SNAP only provide a DHCP interface and a dynamic (non-observable) MAC address (for some reason). As such, we have to now turn on the SNAP, wait for it to try to get an IP address from dnsmasq so we know it's MAC, then update the dhcp-host line and restart the DHCP server. Power cycle the SNAP (or turn it on if it wasn't turned on yet) following the instructions in operation Wait a moment and open the log of dnsmasq with journalctl -u dnsmasq , skip to the bottom with G (Shift + g) You should see a line like Aug 16 14:39:06 grex-caltech-ovro dnsmasq-dhcp[5115]: 1085377743 DHCPDISCOVER(enp1s0f0) 00:40:bf:06:13:02 no address available This implies the SNAP has a MAC address of 00:40:bf:06:13:02 (yours will be different). 4. Go back and uncomment and edit the dhcp-host line of /etc/dnsmasq.conf to contain this MAC. For example, in this case we would put dhcp-host=00:40:bf:06:13:02,192.168.0.3,snap 5. Finally, restart the dhcp server with sudo systemctl restart dnsmasq After waiting a bit for the SNAP to send a new request for a DHCP lease, look at the latest logs again from journalctl. If it ends with something like Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPREQUEST(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 tags: known, enp1s0f0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPACK(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 snap Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 requested options: 1:netmask, 3:router, 28:broadcast, 6:dns-server Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 next server: 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 1 option: 53 message-type 5 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 54 server-identifier 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 51 lease-time 1h Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 58 T1 30m Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 59 T2 52m30s Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 28 broadcast 192.168.0.255 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 1 netmask 255.255.255.0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 3 router 192.168.0.1 That means the SNAP got an IP. You should now be able to ping 192.168.0.3 to make sure it's alive. Advanced 10 GbE Settings Unfortunately, the OS's default configuration for the 10 GbE network card is not optimized for our use-case of streaming time domain science data. As such, we need to adjust a few things. Create the file /etc/sysctl.d/20-grex.conf with the following contents: kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.core.optmem_max = 16777216 vm.swappiness=1 Then apply these changes with sudo sysctl --system Now, we need a program called ethtool to apply some more settings sudo apt-get install ethtool -y Now we will create a file to run on boot to apply a sequence of ethtool settings. Create the file /etc/rc.local with the following contents: #!/bin/env bash ethtool -G enp1s0f0 rx 4096 tx 4096 ethtool -A enp1s0f0 rx on ethtool -A enp1s0f0 tx on Make this file executable with sudo chmod +x /etc/rc.local Now create the file /etc/systemd/system/rc-local.service with the following contents: [Unit] Description = /etc/rc.local Compatibility ConditionPathExists = /etc/rc.local [Service] Type = forking ExecStart = /etc/rc.local start TimeoutSec = 0 StandardOutput = tty RemainAfterExit = yes SysVStartPriority = 99 [Install] WantedBy = multi-user.target Then enable the rc-local service sudo systemctl enable rc-local Finally, reboot GPU Drivers / CUDA Heimdall (the pulse detection part of our pipeline) relies on the CUDA toolkit. Let's install that now (version 12.3) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install cuda-toolkit-12-3 And then install the open-source kernel drivers (Version 545) (overwriting previously installed ones) sudo apt-get install -y nvidia-kernel-open-545 sudo apt-get install -y cuda-drivers-545 You may want to run the following to cleanup old dependencies/driver versions that may have been preinstalled sudo apt-get autoremove Reboot. Then run nvidia-smi to see if the CUDA version and Driver version came up correctly. Finally, add the following to ~/.bashrc to let our use use CUDA # CUDA 12.3 export PATH = /usr/local/cuda-12.3/bin ${ PATH :+: ${ PATH }} export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/cuda-12.3/lib64 and source ~/.bashrc or relog. Warning If you change CUDA versions, you'll need to update these paths! Pipeline Dependencies PSRDADA We use PSRDADA to connect the packet capture and first stage processing pipeline T0 to the pulse detection framework heimdall . This is a library we need to install. We will build a few programs, so might as well create a directory to do this in to keep our home directory organized. mkdir src && cd src Then clone PSRDADA git clone git://git.code.sf.net/p/psrdada/code psrdada && cd psrdada # Last tested version, bump as appropriate git checkout 008afa7 Now, install some build dependencies sudo apt-get install build-essential cmake ninja-build -y Then, build PSRDADA mkdir build && cd build cmake -GNinja .. ninja sudo ninja install This will install the control programs and libraries to /usr/local/bin and /usr/local/lib , respectively. We have to add the latter to out linker path, by adding the following to ~./bashrc # PSRDADA export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/lib Then, relog once agian. Heimdall Similar process to build the pulse-detection software, heimdall. First, clone our fork in our ~/src directory: git clone --recurse-submodules https://github.com/GReX-Telescope/heimdall-astro cd heimdall-astro Install some build dependencies sudo apt-get install libboost-all-dev -y Then build mkdir build && cd build cmake -GNinja .. ninja Run the test dedispersion program to make sure everything seemed to work ./dedisp/testdedisp which should return ----------------------------- INPUT DATA --------------------------------- Frequency of highest chanel (MHz) : 1581.0000 Bandwidth (MHz) : 100.00 NCHANS (Channel Width [MHz]) : 1024 (-0.097656) Sample time (after downsampling by 1) : 0.000250 Observation duration (s) : 30.000000 (119999 samples) Data RMS ( 8 bit input data) : 25.000000 Input data array size : 468 MB Embedding signal ----------------------------- INJECTED SIGNAL ---------------------------- Pulse time at f0 (s) : 3.141590 (sample 12566) Pulse DM (pc/cm^3) : 41.159000 Signal Delays : 0.000000, 0.000008, 0.000017 ... 0.009530 Rawdata Mean (includes signal) : -0.002202 Rawdata StdDev (includes signal) : 25.001451 Pulse S/N (per frequency channel) : 1.000000 Quantizing array Quantized data Mean (includes signal) : 127.497818 Quantized data StdDev (includes signal) : 25.003092 Init GPU Create plan Gen DM list ----------------------------- DM COMPUTATIONS ---------------------------- Computing 32 DMs from 2.000000 to 102.661667 pc/cm^3 Max DM delay is 95 samples (0 seconds) Computing 119904 out of 119999 total samples (99.92% efficiency) Output data array size : 14 MB Compute on GPU Dedispersion took 0.02 seconds Output RMS : 0.376464 Output StdDev : 0.002307 DM trial 11 (37.681 pc/cm^3), Samp 12566 (3.141500 s): 0.390678 (6.16 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12567 (3.141750 s): 0.398160 (9.41 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12568 (3.142000 s): 0.393198 (7.25 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12569 (3.142250 s): 0.391713 (6.61 sigma) DM trial 12 (40.926 pc/cm^3), Samp 12566 (3.141500 s): 0.441719 (28.29 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12564 (3.141000 s): 0.400574 (10.45 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12565 (3.141250 s): 0.403097 (11.55 sigma) Dedispersion successful. Finally, install this into our path with sudo ninja install The heimdall executable should now be available for the pipeline as well as offline analysis. HDF5/NetCDF We use the fantastic netcdf library to write out voltage data. To do this, we need to statically link netcdf (and it's dependent HDF5) at compile time. As such, we have to install these libraries systemw-wide. sudo apt-get install libhdf5-dev libnetcdf-dev -y Rust Many parts of the pipeline software are written in the Rust programming language. We will be building this software from scratch, so we need to install the rust compiler and its tooling. This is easy enough with rustup We need curl for the rustup installer, so sudo apt-get install curl -y Then run the installer, using all the default settings curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Python To get out of version hell for python stuff, we're using Poetry . To install it, we will: curl -sSL https://install.python-poetry.org | python3 - We need to make a few adjustments to ~/.bashrc to correct the paths and fix a bug. Append the following to the end. export PATH = \"/home/user/.local/bin: $PATH \" # Fix the \"Poetry: Failed to unlock the collection\" issue export PYTHON_KEYRING_BACKEND = keyring.backends.null.Keyring Go ahead and source ~/.bashrc now to get these changes in your shell. Pipeline Software To organize all the software needed for running the whole pipeline, we will grab the metapackage from github and clone somewhere (like the home directory): cd git clone --recurse-submodules https://github.com/GReX-Telescope/grex Then, assuming you followed all the previous steps, build the pipeline software with ./grex/build.sh Lastly, you'll need to install the parallel package sudo apt install parallel -y Databases and Metrics Collection Grafana We use Grafana as the observability frontend. Here, we'll assume you are going to set up a Grafana cloud (free-tier) account. While it is possible to use a self-hosted version, we found this to be the most simple. Steps: 1. Create a grafana account here 2. Pick a stack URL that makes sense 3. Click \"Hosted Telemetry Data\" 4. Click \"Hosted Prometheus metrics\" 5. With the default settings of \"via Grafana Alloy\", scroll down to generate an API token (use any name). 6. Record the url , username , and password of the generated Alloy configuration 7. Click the grafana logo in the top left corner and then hit \"Get started\" again. 8. This time, follow \"Hosted Telemetry Data\" with \"OpenTelemetry (OTLP)\" 9. Again, create an API token, scroll down to the generated Alloy configuration file and find the two sections for \"grafana_cloud\" 10. Record the client.endpoint and username and password 11. Again, return to the home screen with the grafana logo, open the menu and navigate to \"Dashboards\" 12. In the top right, go to \"New\" and then \"Import\" 13. Go to our observability stack and either download or copy the dashboard JSON from here 14. Either load or copy the model and hit \"Load\" and then finish the import. This is our main pipeline visualization. 15. Go back to dashboards, import again, but this type supply 1860 for the dashboard ID. This is the \"Node Exporter\" dashboard that will let you monitor all the metrics about your server such as disk usage, network activity, etc. 16. Select our previously setup prometheus data source 17. Now, continue to the docker setup below: Docker Following docker's documentation , install docker the usual way as that is what orchestrates the databases, log aggregation, and communication to grafana. Observability Stack Somewhere obvious (like the home dir), clone the stack . Copy alloy.env.example to alloy.env and fill out the sections according to your grafana configuration. GRAFANA_OTLP_ENDPOINT is client.endpoint from the OpenTelemetry config GRAFANA_OTLP_USER is username from the OpenTelemetry config GRAFANA_OTLP_PASSWORD is username from the OpenTelemetry config GRAFANA_PROM_ENDPOINT is url from the Prometheus config GRAFANA_PROM_USER is the username from the Promethus config GRAFANA_PROM_PASS is the password from the Prometheus config Following the Grafana example above, a completed alloy.env file will look like this Set docker to run as a systemd service with sudo systemctl enable docker if you didn't already in the installation. Test the stack with sudo docker compose up Navigate to your Grafana Node Exporter Full dashboard (it may take a reselect of Datasource to populate) but you should now see the dashboard with all its data. Ctrl-C out of that and then start up the stack in the background with sudo docker compose up -d Pi SSH The last thing will be to configure easy access to the Raspberry Pi's SSH. We can do that by creating a passwordless SSH key and installing it on the pi. We're going to be behind ssh anyway, and the Pi isn't public-facing, so this is a safe thing to do. On the server, generate a new ssh keypair with ssh-keygen -t rsa Then, install it on the pi with ssh-copy-id pi@192.168.0.2 Finally, create an SSH config that automatically supplies the hostname and user: Create a file on the GReX server in ~/.ssh/config with the contents Host pi Hostname 192.168.0.2 User pi Now you can test the easy connection with ssh pi All done! We should now be ready to run the pipeline!","title":"Server Setup"},{"location":"software/server_setup/#server-setup","text":"Once you get your server (either from Puget systems or otherwise), we need to setup additional hardware, adjust some system settings, setup networking, and install the pipeline software.","title":"Server Setup"},{"location":"software/server_setup/#hardware-setup","text":"The server straight from Puget does not have the GPU or 10 GbE network card installed, we will do this first. Open the case and remove the GPU retention bracket Remove the test GPU (T1000), keep this for later in case we need an aditional video output Install the 3090 Ti in the first GPU slot, this will take up three slots of space Install the network card in the bottom slot Wire the 3090 Ti power cable to the harness provided by Puget (they knew this was the GPU we were going to install) Remove the GPU retention clips from the retention bracket that would interfere with the card. It's too tall for it anyway. Replace the retention bracket and close the case. Finally, we need to hook up a monitor to the 3090 Ti so we can setup the software","title":"Hardware Setup"},{"location":"software/server_setup/#initial-os-setup","text":"On boot, you will be presented with the Ubuntu graphical interface. If a password is requested, the deafult is provided in the information booklet that came with the hardware. First, we will change the system password. Set this to something memorable. passwd $( whoami ) Now, we will update the system. There shouldn't be many updates if this is a new machine. sudo apt-get update sudo apt-get upgrade -y We will be editing a bunch of files, if you are comfy in the command line, you probably want to install some editors. Otherwise the graphical gedit tool will be fine. sudo apt-get install emacs vim -y Finally, we will set the hostname. We'll be using the grex-- paradigm (just for clarity, no real reason not to). As in, the first server that is managed by Caltech at OVRO will be grex-caltech-ovro . sudo hostnamectl set-hostname Some updates may require a reboot. If it asks, do that now.","title":"Initial OS Setup"},{"location":"software/server_setup/#networking","text":"Now, we need to setup the networking for the GReX system. We will operate under the assumption that the internet-facing connection will get an IP address from a DHCP server. If that is not the case, consult whoever runs your network on the appropriate setup. Regardless of the WAN connection, the 10 GbE fiber connection to the GReX terminal will be configured the same.","title":"Networking"},{"location":"software/server_setup/#overview","text":"The 10 GbE fiber port serves a few purposes. It is the main data transfer link between the FPGA in the field and the server, but it also carries the monitor and control for the box. This monitor and control connection includes the SNAP control connection and the Raspberry Pi. The SNAP requires an external DHCP server, which we have to provide on this port. Additionally, the 10 GbE switch in the box has its own default subnet for configuration ( 192.168.88.X ). To make everything talk to each other, we need to add two IPs on this port: one in the subnet of the switch's config interface, and the other for DHCP of the various devices.","title":"Overview"},{"location":"software/server_setup/#netplan","text":"In /etc/netplan remove any files that are currently there. Check whether you are using NetworkManager or networkd: systemctl status NetworkManager systemctl status systemd-networkd If NetworkManager is running and networkd is not, disable NetworkManager and enable networkd. (Otherwise, skip this step.) sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo systemctl enable systemd-networkd Then, create a new file called config.yaml with the following contents network : version : 2 renderer : networkd ethernets : # Two WAN interfaces. Configure this according to your network setup enp36s0f0 : dhcp4 : true enp36s0f1 : dhcp4 : true # 10 GbE connection over fiber to the box enp1s0f0 : mtu : 9000 addresses : - 192.168.0.1/24 - 192.168.88.2/24 Then apply with sudo netplan apply","title":"Netplan"},{"location":"software/server_setup/#dhcp-server","text":"Now, we need to setup the DHCP server on the 10 GbE port. First, we install the DHCP server software: sudo apt-get install dnsmasq Create the configuration file in /etc/dnsmasq.conf # Only bind to the 10 GbE interface interface = enp1s0f0 # Disable DNS port = 0 # DHCP Options dhcp-range = 192.168.0.0,static dhcp-option = option:router,192.168.0.1 dhcp-option = option:netmask,255.255.255.0 #dhcp-host=,192.168.0.3,snap log-async log-queries log-dhcp Then, enable the DHCP server service sudo systemctl enable dnsmasq --now This sets up a very simple DHCP server that will give the IP address 192.168.0.3 to the SNAP. Unfortunately, the folks who set up the networking interface for the SNAP only provide a DHCP interface and a dynamic (non-observable) MAC address (for some reason). As such, we have to now turn on the SNAP, wait for it to try to get an IP address from dnsmasq so we know it's MAC, then update the dhcp-host line and restart the DHCP server. Power cycle the SNAP (or turn it on if it wasn't turned on yet) following the instructions in operation Wait a moment and open the log of dnsmasq with journalctl -u dnsmasq , skip to the bottom with G (Shift + g) You should see a line like Aug 16 14:39:06 grex-caltech-ovro dnsmasq-dhcp[5115]: 1085377743 DHCPDISCOVER(enp1s0f0) 00:40:bf:06:13:02 no address available This implies the SNAP has a MAC address of 00:40:bf:06:13:02 (yours will be different). 4. Go back and uncomment and edit the dhcp-host line of /etc/dnsmasq.conf to contain this MAC. For example, in this case we would put dhcp-host=00:40:bf:06:13:02,192.168.0.3,snap 5. Finally, restart the dhcp server with sudo systemctl restart dnsmasq After waiting a bit for the SNAP to send a new request for a DHCP lease, look at the latest logs again from journalctl. If it ends with something like Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPREQUEST(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 tags: known, enp1s0f0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPACK(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 snap Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 requested options: 1:netmask, 3:router, 28:broadcast, 6:dns-server Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 next server: 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 1 option: 53 message-type 5 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 54 server-identifier 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 51 lease-time 1h Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 58 T1 30m Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 59 T2 52m30s Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 28 broadcast 192.168.0.255 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 1 netmask 255.255.255.0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 3 router 192.168.0.1 That means the SNAP got an IP. You should now be able to ping 192.168.0.3 to make sure it's alive.","title":"DHCP Server"},{"location":"software/server_setup/#advanced-10-gbe-settings","text":"Unfortunately, the OS's default configuration for the 10 GbE network card is not optimized for our use-case of streaming time domain science data. As such, we need to adjust a few things. Create the file /etc/sysctl.d/20-grex.conf with the following contents: kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.core.optmem_max = 16777216 vm.swappiness=1 Then apply these changes with sudo sysctl --system Now, we need a program called ethtool to apply some more settings sudo apt-get install ethtool -y Now we will create a file to run on boot to apply a sequence of ethtool settings. Create the file /etc/rc.local with the following contents: #!/bin/env bash ethtool -G enp1s0f0 rx 4096 tx 4096 ethtool -A enp1s0f0 rx on ethtool -A enp1s0f0 tx on Make this file executable with sudo chmod +x /etc/rc.local Now create the file /etc/systemd/system/rc-local.service with the following contents: [Unit] Description = /etc/rc.local Compatibility ConditionPathExists = /etc/rc.local [Service] Type = forking ExecStart = /etc/rc.local start TimeoutSec = 0 StandardOutput = tty RemainAfterExit = yes SysVStartPriority = 99 [Install] WantedBy = multi-user.target Then enable the rc-local service sudo systemctl enable rc-local Finally, reboot","title":"Advanced 10 GbE Settings"},{"location":"software/server_setup/#gpu-drivers-cuda","text":"Heimdall (the pulse detection part of our pipeline) relies on the CUDA toolkit. Let's install that now (version 12.3) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install cuda-toolkit-12-3 And then install the open-source kernel drivers (Version 545) (overwriting previously installed ones) sudo apt-get install -y nvidia-kernel-open-545 sudo apt-get install -y cuda-drivers-545 You may want to run the following to cleanup old dependencies/driver versions that may have been preinstalled sudo apt-get autoremove Reboot. Then run nvidia-smi to see if the CUDA version and Driver version came up correctly. Finally, add the following to ~/.bashrc to let our use use CUDA # CUDA 12.3 export PATH = /usr/local/cuda-12.3/bin ${ PATH :+: ${ PATH }} export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/cuda-12.3/lib64 and source ~/.bashrc or relog. Warning If you change CUDA versions, you'll need to update these paths!","title":"GPU Drivers / CUDA"},{"location":"software/server_setup/#pipeline-dependencies","text":"","title":"Pipeline Dependencies"},{"location":"software/server_setup/#psrdada","text":"We use PSRDADA to connect the packet capture and first stage processing pipeline T0 to the pulse detection framework heimdall . This is a library we need to install. We will build a few programs, so might as well create a directory to do this in to keep our home directory organized. mkdir src && cd src Then clone PSRDADA git clone git://git.code.sf.net/p/psrdada/code psrdada && cd psrdada # Last tested version, bump as appropriate git checkout 008afa7 Now, install some build dependencies sudo apt-get install build-essential cmake ninja-build -y Then, build PSRDADA mkdir build && cd build cmake -GNinja .. ninja sudo ninja install This will install the control programs and libraries to /usr/local/bin and /usr/local/lib , respectively. We have to add the latter to out linker path, by adding the following to ~./bashrc # PSRDADA export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/lib Then, relog once agian.","title":"PSRDADA"},{"location":"software/server_setup/#heimdall","text":"Similar process to build the pulse-detection software, heimdall. First, clone our fork in our ~/src directory: git clone --recurse-submodules https://github.com/GReX-Telescope/heimdall-astro cd heimdall-astro Install some build dependencies sudo apt-get install libboost-all-dev -y Then build mkdir build && cd build cmake -GNinja .. ninja Run the test dedispersion program to make sure everything seemed to work ./dedisp/testdedisp which should return ----------------------------- INPUT DATA --------------------------------- Frequency of highest chanel (MHz) : 1581.0000 Bandwidth (MHz) : 100.00 NCHANS (Channel Width [MHz]) : 1024 (-0.097656) Sample time (after downsampling by 1) : 0.000250 Observation duration (s) : 30.000000 (119999 samples) Data RMS ( 8 bit input data) : 25.000000 Input data array size : 468 MB Embedding signal ----------------------------- INJECTED SIGNAL ---------------------------- Pulse time at f0 (s) : 3.141590 (sample 12566) Pulse DM (pc/cm^3) : 41.159000 Signal Delays : 0.000000, 0.000008, 0.000017 ... 0.009530 Rawdata Mean (includes signal) : -0.002202 Rawdata StdDev (includes signal) : 25.001451 Pulse S/N (per frequency channel) : 1.000000 Quantizing array Quantized data Mean (includes signal) : 127.497818 Quantized data StdDev (includes signal) : 25.003092 Init GPU Create plan Gen DM list ----------------------------- DM COMPUTATIONS ---------------------------- Computing 32 DMs from 2.000000 to 102.661667 pc/cm^3 Max DM delay is 95 samples (0 seconds) Computing 119904 out of 119999 total samples (99.92% efficiency) Output data array size : 14 MB Compute on GPU Dedispersion took 0.02 seconds Output RMS : 0.376464 Output StdDev : 0.002307 DM trial 11 (37.681 pc/cm^3), Samp 12566 (3.141500 s): 0.390678 (6.16 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12567 (3.141750 s): 0.398160 (9.41 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12568 (3.142000 s): 0.393198 (7.25 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12569 (3.142250 s): 0.391713 (6.61 sigma) DM trial 12 (40.926 pc/cm^3), Samp 12566 (3.141500 s): 0.441719 (28.29 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12564 (3.141000 s): 0.400574 (10.45 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12565 (3.141250 s): 0.403097 (11.55 sigma) Dedispersion successful. Finally, install this into our path with sudo ninja install The heimdall executable should now be available for the pipeline as well as offline analysis.","title":"Heimdall"},{"location":"software/server_setup/#hdf5netcdf","text":"We use the fantastic netcdf library to write out voltage data. To do this, we need to statically link netcdf (and it's dependent HDF5) at compile time. As such, we have to install these libraries systemw-wide. sudo apt-get install libhdf5-dev libnetcdf-dev -y","title":"HDF5/NetCDF"},{"location":"software/server_setup/#rust","text":"Many parts of the pipeline software are written in the Rust programming language. We will be building this software from scratch, so we need to install the rust compiler and its tooling. This is easy enough with rustup We need curl for the rustup installer, so sudo apt-get install curl -y Then run the installer, using all the default settings curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh","title":"Rust"},{"location":"software/server_setup/#python","text":"To get out of version hell for python stuff, we're using Poetry . To install it, we will: curl -sSL https://install.python-poetry.org | python3 - We need to make a few adjustments to ~/.bashrc to correct the paths and fix a bug. Append the following to the end. export PATH = \"/home/user/.local/bin: $PATH \" # Fix the \"Poetry: Failed to unlock the collection\" issue export PYTHON_KEYRING_BACKEND = keyring.backends.null.Keyring Go ahead and source ~/.bashrc now to get these changes in your shell.","title":"Python"},{"location":"software/server_setup/#pipeline-software","text":"To organize all the software needed for running the whole pipeline, we will grab the metapackage from github and clone somewhere (like the home directory): cd git clone --recurse-submodules https://github.com/GReX-Telescope/grex Then, assuming you followed all the previous steps, build the pipeline software with ./grex/build.sh Lastly, you'll need to install the parallel package sudo apt install parallel -y","title":"Pipeline Software"},{"location":"software/server_setup/#databases-and-metrics-collection","text":"","title":"Databases and Metrics Collection"},{"location":"software/server_setup/#grafana","text":"We use Grafana as the observability frontend. Here, we'll assume you are going to set up a Grafana cloud (free-tier) account. While it is possible to use a self-hosted version, we found this to be the most simple. Steps: 1. Create a grafana account here 2. Pick a stack URL that makes sense 3. Click \"Hosted Telemetry Data\" 4. Click \"Hosted Prometheus metrics\" 5. With the default settings of \"via Grafana Alloy\", scroll down to generate an API token (use any name). 6. Record the url , username , and password of the generated Alloy configuration 7. Click the grafana logo in the top left corner and then hit \"Get started\" again. 8. This time, follow \"Hosted Telemetry Data\" with \"OpenTelemetry (OTLP)\" 9. Again, create an API token, scroll down to the generated Alloy configuration file and find the two sections for \"grafana_cloud\" 10. Record the client.endpoint and username and password 11. Again, return to the home screen with the grafana logo, open the menu and navigate to \"Dashboards\" 12. In the top right, go to \"New\" and then \"Import\" 13. Go to our observability stack and either download or copy the dashboard JSON from here 14. Either load or copy the model and hit \"Load\" and then finish the import. This is our main pipeline visualization. 15. Go back to dashboards, import again, but this type supply 1860 for the dashboard ID. This is the \"Node Exporter\" dashboard that will let you monitor all the metrics about your server such as disk usage, network activity, etc. 16. Select our previously setup prometheus data source 17. Now, continue to the docker setup below:","title":"Grafana"},{"location":"software/server_setup/#docker","text":"Following docker's documentation , install docker the usual way as that is what orchestrates the databases, log aggregation, and communication to grafana.","title":"Docker"},{"location":"software/server_setup/#observability-stack","text":"Somewhere obvious (like the home dir), clone the stack . Copy alloy.env.example to alloy.env and fill out the sections according to your grafana configuration. GRAFANA_OTLP_ENDPOINT is client.endpoint from the OpenTelemetry config GRAFANA_OTLP_USER is username from the OpenTelemetry config GRAFANA_OTLP_PASSWORD is username from the OpenTelemetry config GRAFANA_PROM_ENDPOINT is url from the Prometheus config GRAFANA_PROM_USER is the username from the Promethus config GRAFANA_PROM_PASS is the password from the Prometheus config Following the Grafana example above, a completed alloy.env file will look like this Set docker to run as a systemd service with sudo systemctl enable docker if you didn't already in the installation. Test the stack with sudo docker compose up Navigate to your Grafana Node Exporter Full dashboard (it may take a reselect of Datasource to populate) but you should now see the dashboard with all its data. Ctrl-C out of that and then start up the stack in the background with sudo docker compose up -d","title":"Observability Stack"},{"location":"software/server_setup/#pi-ssh","text":"The last thing will be to configure easy access to the Raspberry Pi's SSH. We can do that by creating a passwordless SSH key and installing it on the pi. We're going to be behind ssh anyway, and the Pi isn't public-facing, so this is a safe thing to do. On the server, generate a new ssh keypair with ssh-keygen -t rsa Then, install it on the pi with ssh-copy-id pi@192.168.0.2 Finally, create an SSH config that automatically supplies the hostname and user: Create a file on the GReX server in ~/.ssh/config with the contents Host pi Hostname 192.168.0.2 User pi Now you can test the easy connection with ssh pi All done! We should now be ready to run the pipeline!","title":"Pi SSH"},{"location":"software/snap/","text":"SNAP Configuration and Bringup The digital backend to the GReX system is a SNAP board from the CASPER group at Berkeley. This board contains the analog to digital converters and Xilinx FPGA to perform the digitization and F-engine components of the system. The setup and configuration of this board has seemingly never been well documented, so we'll try to make it as painless as possible here. c The FPGA Simulink model is stored here with the latest releases found here . Grab the latest fpg file, and you're good to go - no reason to recompile it. SNAP Golden Image To document the steps that are performed before shipping a box, here are the steps to program the golden (bootstrapping) image: Set switch S1 on the SANP so that switches 2 and 5 are set to on. (In the on position, the switches are moved towards the edge of the PCB). The other switches on S1 should be off. This allows the SNAP to boot from the nonvolatile flash memory. To flash this storage device, you need the free Vivado Lab Edition (VLE) and the expensive Xilinx Platform Cable. After launching VLE, click Open hardware manager . After this, a green bar at the top should ask you to autoconnect. Assuming the platform cable is plugged into the SNAP and everything is powered on, click on that. It should connect and show the xc7k160t FPGA in the hardware section on the left. In VLE, under tools go to Add configuration memory device . Use the memory device n25q256-3.3v-spi-x1_x2_x4 . It will then ask if you want to program the memory now, hit Ok . Using the golden image for the configuration file here , program the entire configuration memory device. Unplug programmer before rebooting. All done! A reboot should start blinking some LEDs and the ethernet should get a DHCP address (if it's plugged in). TAPCP and the Raspberry Pi The gateware we've built for the SNAP board includes a small CPU called the MicroBlaze that hosts a small webserver to deal with runtime interactions with FPGA memory. This server gets an IP address from a DHCP server running on the main capture computer. This interface can also be used to reprogram the SNAP if the gateware changes. By deafult, we'll ship SNAP boards that have the GReX gateware preprogramed, but it's always possible to reprogram it. This interface is over the UDP protocol TFTP, where the folks at CASPER have wrapped reading and writing files as the interaction to both the flash and FPU memory. We've written a wrapper to the so called \"TAPCP\" protocol here . It is with this library that the packet capture code informs the SNAP when to activate timing signals. FPGA Clocks, References, PPS, Synthesizers The FPGA needs an external clock source, which is being provided via one of the outputs of the Valon synthesizer in the box. Additionally, the board needs \"pulse per second\" (PPS) ticks, which come from the GPS reciever. The Valon synthesizer has two outputs but can lock both to an external reference. The GPS receiver in the box provides this (10 MHz) alongside the PPS signal. SNAP Operation Controlling the Gateware UDP Payloads Once the pipeline is running, the SNAP will be streaming high-speed data over UDP to the processing server. These UDP frames contain one timestep of both polarizations in 8+8 bit complex data. Additionally, the payload starts with a 64-bit header that is the number of frames since the first one. Assuming you record the time when you trigger the start of packets, you can then work out the associated timestamp of the packet using the fact that every packet arrives in a 8.192us cadence. The format of the payload itself has a C struct compatible layout. +-----------------------------------------------------------------------+ | Timestamp (unsigned 64-bit, little-endian integer) | +-----------------------------------------------------------------------+ | A0000R | A0000I | A0001R | A0001I | A0002R | A0002I | A0003R | A0003I | | ... | | A2044R | A2044I | A2045R | A2045I | A2046R | A2046I | A2047R | A2047I | +-----------------------------------------------------------------------+ | B0000R | B0000I | B0001R | B0001I | B0002R | B0002I | B0003R | B0003I | | ... | | B2044R | B2044I | B2045R | B2045I | B2046R | B2046I | B2047R | B2047I | +-----------------------------------------------------------------------+ Where XnnnnR + XnnnnI *j is the complex voltage data of polarization X (either A or B), channel nnnn where 0000 is 1530 MHz and 2047 is 1280 MHz. The components are signed bytes.","title":"SNAP"},{"location":"software/snap/#snap-configuration-and-bringup","text":"The digital backend to the GReX system is a SNAP board from the CASPER group at Berkeley. This board contains the analog to digital converters and Xilinx FPGA to perform the digitization and F-engine components of the system. The setup and configuration of this board has seemingly never been well documented, so we'll try to make it as painless as possible here. c The FPGA Simulink model is stored here with the latest releases found here . Grab the latest fpg file, and you're good to go - no reason to recompile it.","title":"SNAP Configuration and Bringup"},{"location":"software/snap/#snap-golden-image","text":"To document the steps that are performed before shipping a box, here are the steps to program the golden (bootstrapping) image: Set switch S1 on the SANP so that switches 2 and 5 are set to on. (In the on position, the switches are moved towards the edge of the PCB). The other switches on S1 should be off. This allows the SNAP to boot from the nonvolatile flash memory. To flash this storage device, you need the free Vivado Lab Edition (VLE) and the expensive Xilinx Platform Cable. After launching VLE, click Open hardware manager . After this, a green bar at the top should ask you to autoconnect. Assuming the platform cable is plugged into the SNAP and everything is powered on, click on that. It should connect and show the xc7k160t FPGA in the hardware section on the left. In VLE, under tools go to Add configuration memory device . Use the memory device n25q256-3.3v-spi-x1_x2_x4 . It will then ask if you want to program the memory now, hit Ok . Using the golden image for the configuration file here , program the entire configuration memory device. Unplug programmer before rebooting. All done! A reboot should start blinking some LEDs and the ethernet should get a DHCP address (if it's plugged in).","title":"SNAP Golden Image"},{"location":"software/snap/#tapcp-and-the-raspberry-pi","text":"The gateware we've built for the SNAP board includes a small CPU called the MicroBlaze that hosts a small webserver to deal with runtime interactions with FPGA memory. This server gets an IP address from a DHCP server running on the main capture computer. This interface can also be used to reprogram the SNAP if the gateware changes. By deafult, we'll ship SNAP boards that have the GReX gateware preprogramed, but it's always possible to reprogram it. This interface is over the UDP protocol TFTP, where the folks at CASPER have wrapped reading and writing files as the interaction to both the flash and FPU memory. We've written a wrapper to the so called \"TAPCP\" protocol here . It is with this library that the packet capture code informs the SNAP when to activate timing signals.","title":"TAPCP and the Raspberry Pi"},{"location":"software/snap/#fpga-clocks-references-pps-synthesizers","text":"The FPGA needs an external clock source, which is being provided via one of the outputs of the Valon synthesizer in the box. Additionally, the board needs \"pulse per second\" (PPS) ticks, which come from the GPS reciever. The Valon synthesizer has two outputs but can lock both to an external reference. The GPS receiver in the box provides this (10 MHz) alongside the PPS signal.","title":"FPGA Clocks, References, PPS, Synthesizers"},{"location":"software/snap/#snap-operation","text":"","title":"SNAP Operation"},{"location":"software/snap/#controlling-the-gateware","text":"","title":"Controlling the Gateware"},{"location":"software/snap/#udp-payloads","text":"Once the pipeline is running, the SNAP will be streaming high-speed data over UDP to the processing server. These UDP frames contain one timestep of both polarizations in 8+8 bit complex data. Additionally, the payload starts with a 64-bit header that is the number of frames since the first one. Assuming you record the time when you trigger the start of packets, you can then work out the associated timestamp of the packet using the fact that every packet arrives in a 8.192us cadence. The format of the payload itself has a C struct compatible layout. +-----------------------------------------------------------------------+ | Timestamp (unsigned 64-bit, little-endian integer) | +-----------------------------------------------------------------------+ | A0000R | A0000I | A0001R | A0001I | A0002R | A0002I | A0003R | A0003I | | ... | | A2044R | A2044I | A2045R | A2045I | A2046R | A2046I | A2047R | A2047I | +-----------------------------------------------------------------------+ | B0000R | B0000I | B0001R | B0001I | B0002R | B0002I | B0003R | B0003I | | ... | | B2044R | B2044I | B2045R | B2045I | B2046R | B2046I | B2047R | B2047I | +-----------------------------------------------------------------------+ Where XnnnnR + XnnnnI *j is the complex voltage data of polarization X (either A or B), channel nnnn where 0000 is 1530 MHz and 2047 is 1280 MHz. The components are signed bytes.","title":"UDP Payloads"},{"location":"software/stack/","text":"Software Stack There are several moving parts to this whole project, most of which are organized in our GitHub organization . The notable pieces of software are: snap_bringup - SNAP bringup and configuration T0 - UDP Packet capture and exfil to heimdal heimdall (T1) - Our fork of the pulse detection pipeline which removes clustering and RFI excision T2 FrontendModule - Hardware and software design for the Frontend Module (FEM) These are supported by some fundamental libraries sigproc_filterbank - A rust library for reading/writing SIGPROC filterbank files psrdada-rs - A rust library for interacting with PSRDADA buffers casperfpga_rs - A rust library for interacting with the SNAP board over TAPCP","title":"Software Stack"},{"location":"software/stack/#software-stack","text":"There are several moving parts to this whole project, most of which are organized in our GitHub organization . The notable pieces of software are: snap_bringup - SNAP bringup and configuration T0 - UDP Packet capture and exfil to heimdal heimdall (T1) - Our fork of the pulse detection pipeline which removes clustering and RFI excision T2 FrontendModule - Hardware and software design for the Frontend Module (FEM) These are supported by some fundamental libraries sigproc_filterbank - A rust library for reading/writing SIGPROC filterbank files psrdada-rs - A rust library for interacting with PSRDADA buffers casperfpga_rs - A rust library for interacting with the SNAP board over TAPCP","title":"Software Stack"}]} \ No newline at end of file +{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"The Global Radio Explorer Telescope Welcome to the documentation for the Global Radio Explorer Telescope! In the tabs above, you'll find the documentation for the hardware and software architectures as well as guides for construction and setup. Status OVRO - Up and Running! Dashboard","title":"Home"},{"location":"#the-global-radio-explorer-telescope","text":"Welcome to the documentation for the Global Radio Explorer Telescope! In the tabs above, you'll find the documentation for the hardware and software architectures as well as guides for construction and setup.","title":"The Global Radio Explorer Telescope"},{"location":"#status","text":"OVRO - Up and Running! Dashboard","title":"Status"},{"location":"about/","text":"The Global Radio Telescope is an new low-cost radio telescope designed to be an all-sky monitor for bright radio bursts. Building on the success of STARE2, we will search for fast radio bursts (FRBs) emitted from Galactic magnetars as well as bursts from nearby galaxies. GReX will search down to ten microseconds time resolution, allowing us to find new super giant radio pulses from Milky Way pulsars and study their broadband emission. The proposed instrument will employ ultra-wide band (0.7-2 GHz) feeds coupled to a high performance (receiver temperature 10 K) low noise amplifier (LNA) originally developed for the DSA-110 and DSA-2000 projects. In GReX Phase I (GReX-I), unit systems will be deployed at Owens Valley Radio Observatory (OVRO) and Big Smoky Valley, Nevada. Phase II will expand the array, placing feeds in India, Australia, and elsewhere in order to build up to continuous coverage of nearly 4\u03c0 steradians and to increase our exposure to the Galactic plane. We model the local magnetar population to forecast for GReX, finding the improved sensitivity and increased exposure to the Galactic plane could lead to dozens of FRB-like bursts per year. For more information, read our paper here ! Team Project Scientist - Dr. Liam Connor, PhD Project Engineer - Kiran Shila, MSEE","title":"About"},{"location":"about/#team","text":"Project Scientist - Dr. Liam Connor, PhD Project Engineer - Kiran Shila, MSEE","title":"Team"},{"location":"hardware/assembly/","text":"Assembly Guide TODO! Once we have the complete pacakge, write a step by step guide for assembly and installation","title":"Assembly Guide"},{"location":"hardware/assembly/#assembly-guide","text":"TODO! Once we have the complete pacakge, write a step by step guide for assembly and installation","title":"Assembly Guide"},{"location":"hardware/box/","text":"The Box Here one can find documentation re: the GReX box, including cables, routing, etc. A 3D PDF is available for download here . The BOM is available here Interface Plate The custom interface plate is machined from stock aluminum measuring approximately 12x3x1/8 inches. The drilled hole specifications are shown in the drawing below: Interface Plate Drawing Tips for machining The drilled holes are measured such that the stock size does not need to be precise. This plate will later be used to mark the corresponding holes on the main box, so the outer eight holes could be measured from all four sides. We assume the long edges arrive machined parallel. Note that the eight tapped holes around the perimeter of the piece should first be drilled with a #29 bit. After aligning the plate and marking the bottom of the box, the holes are re-drilled using the specified #18 bit. To adhere the interface plate to the main box, first the cutout must be made with a jig saw. Refer to the drawing below for placement. Note that the \"bottom\" of the box is defined when the lid is facing upwards and the hinge is on the left side. Double check that the internal ground pin is not removed or damaged with the cutout. Box Cutout Drawing Tips for machining After marking the box according to the above drawing, we marked four points: one inside each corner of the cutout, offset 3/8\" from the edge. We drilled 3/4\" holes at each point, which served as entry points for the jigsaw blade (20 TPI, no oscillation). File generously to remove sharp edges. After the cutout is finished, use the interface plate to mark the location of the eight screw holes. When determining the position of the interface plate, ensure that there is sufficient clearance space for the locking washer on the nipple pipe. After making the bottom cutout, all external surfaces of the box are painted white. Affix the interface plate and conductive rubber gasket to the box with pan head cross screws, #8-32 5/16\", containing a rubber O-ring. Interface Plate Assembly Grind down the back cover of the weatherproof box so that it lays flat against the interface plate. Drill the center hole with the step drill and the clearance holes for the 8-32 screws. Alignment is not critical, as long as the center feedthrough connectors are within the center hole, and the distance between the outer tapped holes matches with their distance on the interface plate. Weatherbox Modifications Note the location of the green ground pin when the weather box is affixed to the interface plate. Use the provided plugs to close both side holes. Punch out the center hole and corresponding screw holes from the provided foam weatherbox gasket. Be sure to thread the extension cord wire through the lid and gasket prior to soldering the wires to the feedthroughs. Close both weatherbox lid side holes with the provided plugs. Weatherbox Assembly The 1in diameter pipe is secured on both sides by a locking washer (not pictured). The three SMAs are secured with one nut outside, one locking washer and two nuts inside. Trim the foam gasket flush to the weatherbox, and the conductive gasket flush to the interface plate. Using RTV108 translucent adhesive, seal the following edges: between the weatherbox lid and body, around all plugs in the weatherbox, around the outside of the interface plate, and around the outer locking washer of the 1in nipple pipe. Enclosure Plate The steel plate that comes provided with the enclosure box is discarded and replaced instead with an aluminum plate. Use a step drill for the four corner post holes. Drill the chassis holes on the bottom layer of the FEM assembly so that they fit 4-40 hardware. For the SNAP and GPS boards which are attached with standoffs, the screw length to affix the board to the standoff is 3/16\" and to affix the standoffs to the plate the length is 1/4\". Enclosure Plate Drawing After affixing the FEM, screw the MiniCircuits amplifiers to the SMA connectors to determine how high they sit off the plate. Fill the gap with 2-56 size washers (should be around 3). Drill the 2-56 clearance holes for the amplifiers with them in place. Choose appropriate screw length, will vary if many more or less than 3 washers used. Enclosure Lid Enclosure Lid Drawing For the RPi and PSU, the standoffs are 4-40x1/2\" hex. Screws through the boards are 3/16\" and screws through the lid are 1/4\". The switch is secured by 5/16\" screws and is backed with a nut. The LRS-50 uses metric hardware 6mm in length. Fan The fan can be attached to the lid with 1/4\" L brackets and 4-40 hardware. 5/16\" screws from the fan into the threaded side of the L bracket. 3/4\" hardware with locking washer + nut combo from L bracket through the lid. In the case where the SNAP FPGA fan is missing, the large fan is mounted directly to the SNAP board, using the three unused mounting holes on the right hand side. The .stl file for the adjustable height 3D printed bracket can be found here . Use #6 hardware to attach the fan to the base, and #2 hardware to attach the base to the SNAP. The fan is wired to the Fan terminal of the PSU using the braided red and white wire. Wiring Wiring Diagram Wiring Key * Valon * Red/white braided wire from +6Vdc to 'Valon' terminal of PSU * 086-3SMR+ cable from Source 1 SMA to top middle FEM * 086-8SMR+ cable from Ext Ref SMA to top left GPS * 086-12SMRSM+ cable from Source 2 SMA to 4th from right SNAP * FEM * 086-24SMRSM+ cable from top left to POL A on interface * 086-24SMRSM+ cable from top right to POL B on interface * 4 port terminal (splice to red/white braided wire) to FEM terminal of PSU * 4 port terminal (splice to blue/green wire) to TXD, RXD terminal of RPi * 086-4 or 086-3SM+ cable from bottom left amp to 6th from left SNAP * Red/white braided wire from bottom left amp to RPi terminal PSU * 086-4 or 086-3SM+ cable from bottom right amp to 9th from left SNAP * Red/white braided wire from bottom right amp to RPi terminal PSU * GPS * 086-15SMRSM+ cable from top right to 2nd from right SNAP * 086-15SMRSM+ cable from bottom SMA to GPS on interface * Power plug to Fan terminal of PSU * SNAP * 18 gauge red/black pair from 6 pin power plug to SNAP terminal of PSU * P1, P2 to SFP+ 1,2 on Switch * Switch * Power plug to Switch terminal of PSU * Ethernet cable from PoE In to bottom left ethernet plug of RPi * LRS-50 * 18 gauge red/black pair from V-, V+ to GND, 12V terminal of PSU * Black, white wire (16 gauge bundle) from L, N to H, N feedthroughs on interface * Green cable (16 gauge bundle) from GND to bottom ground pin * 16 gauge green cable from lid ground pin to bottom ground pin * RPi * Red/white braided wire from 5V, GND terminal to RPi terminal of PSU * Red/white braided wire from IO20, GND terminal to SW, GND of PSU Mounting Brackets Cut the square tube to 15\" plus four 1\" spacers. Using the 'X' size drill bit, drill out the 10th hole and drill a hole between the 6th and 7th holes from the uncut end. Insert the v-bolt and saddle lock into the drilled holes, fix with washer and nut. Insert bolts through the third hole in from each end. Add 1\" spacer to each, and thread through the tabs on the back of the box. Affix with a washer and nut. Mounting Brackets Final Steps Apply lock-tite to all hardware except those with nylocks. Run fiber cable through the 1 in nipple pipe into the Switch. Plug pipe with steel wool. Screw feed to the top of the box using 8/32 hex screws and nuts. The LNAs connect to the terminals on the feed, and SMA cables connect the LNA outputs to POLA and POLB on the interface plate. The magnetic GPS unit attaches to a side arm, and screws into the GPS port on the interface plate. TODO: side arm? The GReX box is designed to mount on a vertical pipe. Use the V bolt and saddle lock to adjust and tighten.","title":"The Box"},{"location":"hardware/box/#the-box","text":"Here one can find documentation re: the GReX box, including cables, routing, etc. A 3D PDF is available for download here . The BOM is available here","title":"The Box"},{"location":"hardware/box/#interface-plate","text":"The custom interface plate is machined from stock aluminum measuring approximately 12x3x1/8 inches. The drilled hole specifications are shown in the drawing below: Interface Plate Drawing Tips for machining The drilled holes are measured such that the stock size does not need to be precise. This plate will later be used to mark the corresponding holes on the main box, so the outer eight holes could be measured from all four sides. We assume the long edges arrive machined parallel. Note that the eight tapped holes around the perimeter of the piece should first be drilled with a #29 bit. After aligning the plate and marking the bottom of the box, the holes are re-drilled using the specified #18 bit. To adhere the interface plate to the main box, first the cutout must be made with a jig saw. Refer to the drawing below for placement. Note that the \"bottom\" of the box is defined when the lid is facing upwards and the hinge is on the left side. Double check that the internal ground pin is not removed or damaged with the cutout. Box Cutout Drawing Tips for machining After marking the box according to the above drawing, we marked four points: one inside each corner of the cutout, offset 3/8\" from the edge. We drilled 3/4\" holes at each point, which served as entry points for the jigsaw blade (20 TPI, no oscillation). File generously to remove sharp edges. After the cutout is finished, use the interface plate to mark the location of the eight screw holes. When determining the position of the interface plate, ensure that there is sufficient clearance space for the locking washer on the nipple pipe. After making the bottom cutout, all external surfaces of the box are painted white. Affix the interface plate and conductive rubber gasket to the box with pan head cross screws, #8-32 5/16\", containing a rubber O-ring. Interface Plate Assembly Grind down the back cover of the weatherproof box so that it lays flat against the interface plate. Drill the center hole with the step drill and the clearance holes for the 8-32 screws. Alignment is not critical, as long as the center feedthrough connectors are within the center hole, and the distance between the outer tapped holes matches with their distance on the interface plate. Weatherbox Modifications Note the location of the green ground pin when the weather box is affixed to the interface plate. Use the provided plugs to close both side holes. Punch out the center hole and corresponding screw holes from the provided foam weatherbox gasket. Be sure to thread the extension cord wire through the lid and gasket prior to soldering the wires to the feedthroughs. Close both weatherbox lid side holes with the provided plugs. Weatherbox Assembly The 1in diameter pipe is secured on both sides by a locking washer (not pictured). The three SMAs are secured with one nut outside, one locking washer and two nuts inside. Trim the foam gasket flush to the weatherbox, and the conductive gasket flush to the interface plate. Using RTV108 translucent adhesive, seal the following edges: between the weatherbox lid and body, around all plugs in the weatherbox, around the outside of the interface plate, and around the outer locking washer of the 1in nipple pipe.","title":"Interface Plate"},{"location":"hardware/box/#enclosure-plate","text":"The steel plate that comes provided with the enclosure box is discarded and replaced instead with an aluminum plate. Use a step drill for the four corner post holes. Drill the chassis holes on the bottom layer of the FEM assembly so that they fit 4-40 hardware. For the SNAP and GPS boards which are attached with standoffs, the screw length to affix the board to the standoff is 3/16\" and to affix the standoffs to the plate the length is 1/4\". Enclosure Plate Drawing After affixing the FEM, screw the MiniCircuits amplifiers to the SMA connectors to determine how high they sit off the plate. Fill the gap with 2-56 size washers (should be around 3). Drill the 2-56 clearance holes for the amplifiers with them in place. Choose appropriate screw length, will vary if many more or less than 3 washers used.","title":"Enclosure Plate"},{"location":"hardware/box/#enclosure-lid","text":"Enclosure Lid Drawing For the RPi and PSU, the standoffs are 4-40x1/2\" hex. Screws through the boards are 3/16\" and screws through the lid are 1/4\". The switch is secured by 5/16\" screws and is backed with a nut. The LRS-50 uses metric hardware 6mm in length.","title":"Enclosure Lid"},{"location":"hardware/box/#fan","text":"The fan can be attached to the lid with 1/4\" L brackets and 4-40 hardware. 5/16\" screws from the fan into the threaded side of the L bracket. 3/4\" hardware with locking washer + nut combo from L bracket through the lid. In the case where the SNAP FPGA fan is missing, the large fan is mounted directly to the SNAP board, using the three unused mounting holes on the right hand side. The .stl file for the adjustable height 3D printed bracket can be found here . Use #6 hardware to attach the fan to the base, and #2 hardware to attach the base to the SNAP. The fan is wired to the Fan terminal of the PSU using the braided red and white wire.","title":"Fan"},{"location":"hardware/box/#wiring","text":"Wiring Diagram Wiring Key * Valon * Red/white braided wire from +6Vdc to 'Valon' terminal of PSU * 086-3SMR+ cable from Source 1 SMA to top middle FEM * 086-8SMR+ cable from Ext Ref SMA to top left GPS * 086-12SMRSM+ cable from Source 2 SMA to 4th from right SNAP * FEM * 086-24SMRSM+ cable from top left to POL A on interface * 086-24SMRSM+ cable from top right to POL B on interface * 4 port terminal (splice to red/white braided wire) to FEM terminal of PSU * 4 port terminal (splice to blue/green wire) to TXD, RXD terminal of RPi * 086-4 or 086-3SM+ cable from bottom left amp to 6th from left SNAP * Red/white braided wire from bottom left amp to RPi terminal PSU * 086-4 or 086-3SM+ cable from bottom right amp to 9th from left SNAP * Red/white braided wire from bottom right amp to RPi terminal PSU * GPS * 086-15SMRSM+ cable from top right to 2nd from right SNAP * 086-15SMRSM+ cable from bottom SMA to GPS on interface * Power plug to Fan terminal of PSU * SNAP * 18 gauge red/black pair from 6 pin power plug to SNAP terminal of PSU * P1, P2 to SFP+ 1,2 on Switch * Switch * Power plug to Switch terminal of PSU * Ethernet cable from PoE In to bottom left ethernet plug of RPi * LRS-50 * 18 gauge red/black pair from V-, V+ to GND, 12V terminal of PSU * Black, white wire (16 gauge bundle) from L, N to H, N feedthroughs on interface * Green cable (16 gauge bundle) from GND to bottom ground pin * 16 gauge green cable from lid ground pin to bottom ground pin * RPi * Red/white braided wire from 5V, GND terminal to RPi terminal of PSU * Red/white braided wire from IO20, GND terminal to SW, GND of PSU","title":"Wiring"},{"location":"hardware/box/#mounting-brackets","text":"Cut the square tube to 15\" plus four 1\" spacers. Using the 'X' size drill bit, drill out the 10th hole and drill a hole between the 6th and 7th holes from the uncut end. Insert the v-bolt and saddle lock into the drilled holes, fix with washer and nut. Insert bolts through the third hole in from each end. Add 1\" spacer to each, and thread through the tabs on the back of the box. Affix with a washer and nut. Mounting Brackets","title":"Mounting Brackets"},{"location":"hardware/box/#final-steps","text":"Apply lock-tite to all hardware except those with nylocks. Run fiber cable through the 1 in nipple pipe into the Switch. Plug pipe with steel wool. Screw feed to the top of the box using 8/32 hex screws and nuts. The LNAs connect to the terminals on the feed, and SMA cables connect the LNA outputs to POLA and POLB on the interface plate. The magnetic GPS unit attaches to a side arm, and screws into the GPS port on the interface plate. TODO: side arm? The GReX box is designed to mount on a vertical pipe. Use the V bolt and saddle lock to adjust and tighten.","title":"Final Steps"},{"location":"hardware/feed/","text":"Feed Antenna TODO! Jonas info","title":"Feed Antenna"},{"location":"hardware/feed/#feed-antenna","text":"TODO! Jonas info","title":"Feed Antenna"},{"location":"hardware/fem/","text":"Frontend Module The frontend module (FEM) is a device that performs the analog signal processing after the LNAs. This includes filtering, downconversion, and amplification. Additionally, this module provides rudimentary monitor and control support. Bare PCB Completed Module Hardware Design The hardware design itself is implemented in the free KiCAD program and is available here . The current hardware uses ENIG to help reflow of the fine-pitch components. Additionally, it utilizes the low-loss RO4003C substrate in a custom 4-layer stackup. Schematics BOM Case Here is an annotated view of the PCB with major system components labeled: Software Design The RF hardware mostly operates without the intervention of any software. The only step required to use the RF hardware is to set the valid attenuation level, which defaults to 0 dB. As such, the primary goal of the digital section of the FEM is to perform Monitor and Control (MnC). MnC is achieved via an 115200 baud 3.3V UART interface on the main connector. The firmware design is carried out in the Rust programming language, and whose source can be found here . In that repository, one can also find the control software, which is used to perform the monitoring and controlling. With this software, there is control for enabling/disabling the LNA bias and setting the interstage IF attenuator. For the digital attenuator, we can set attenuation (and therefore change the total gain) from 0 dB to 31.5 dB. This attenuator is used to maximize the dynamic range of the ADC and can be set for environmental RFI levels. CLI To use the CLI program, connect the FEM to a serial port either with a USB to serial adapter or directly to a hardware UART port (like on the Raspberry Pi). If the pi doesn't have the cli program already copied over, it can be found here . Usage: cli Commands: mon Gets monitor data from the FEM lna Controls the power of the LNA if Sets the IF \"power good\" threshold atten Sets the attenuation level in dB ( 0 to 31 .5 ) help Print this message or the help of the given subcommand ( s ) Arguments: Serial port for the FEM Options: -h, --help Print help -V, --version Print version To set the power state of the LNAs, use: ./cli lna ch< 1 | 2 > To set the attenuation: ./cli atten < 0 -31.5> To grab the monitor data: ./cli mon Physical Interface There are eight LEDs on the front panel. Four red LEDs to indicate power statess, two blue LEDs for serial activity, and two green LEDs for system status. The green LEDs will be enabled when the IF power is at a nominal level.","title":"Fronend Module"},{"location":"hardware/fem/#frontend-module","text":"The frontend module (FEM) is a device that performs the analog signal processing after the LNAs. This includes filtering, downconversion, and amplification. Additionally, this module provides rudimentary monitor and control support. Bare PCB Completed Module","title":"Frontend Module"},{"location":"hardware/fem/#hardware-design","text":"The hardware design itself is implemented in the free KiCAD program and is available here . The current hardware uses ENIG to help reflow of the fine-pitch components. Additionally, it utilizes the low-loss RO4003C substrate in a custom 4-layer stackup. Schematics BOM Case Here is an annotated view of the PCB with major system components labeled:","title":"Hardware Design"},{"location":"hardware/fem/#software-design","text":"The RF hardware mostly operates without the intervention of any software. The only step required to use the RF hardware is to set the valid attenuation level, which defaults to 0 dB. As such, the primary goal of the digital section of the FEM is to perform Monitor and Control (MnC). MnC is achieved via an 115200 baud 3.3V UART interface on the main connector. The firmware design is carried out in the Rust programming language, and whose source can be found here . In that repository, one can also find the control software, which is used to perform the monitoring and controlling. With this software, there is control for enabling/disabling the LNA bias and setting the interstage IF attenuator. For the digital attenuator, we can set attenuation (and therefore change the total gain) from 0 dB to 31.5 dB. This attenuator is used to maximize the dynamic range of the ADC and can be set for environmental RFI levels.","title":"Software Design"},{"location":"hardware/fem/#cli","text":"To use the CLI program, connect the FEM to a serial port either with a USB to serial adapter or directly to a hardware UART port (like on the Raspberry Pi). If the pi doesn't have the cli program already copied over, it can be found here . Usage: cli Commands: mon Gets monitor data from the FEM lna Controls the power of the LNA if Sets the IF \"power good\" threshold atten Sets the attenuation level in dB ( 0 to 31 .5 ) help Print this message or the help of the given subcommand ( s ) Arguments: Serial port for the FEM Options: -h, --help Print help -V, --version Print version To set the power state of the LNAs, use: ./cli lna ch< 1 | 2 > To set the attenuation: ./cli atten < 0 -31.5> To grab the monitor data: ./cli mon","title":"CLI"},{"location":"hardware/fem/#physical-interface","text":"There are eight LEDs on the front panel. Four red LEDs to indicate power statess, two blue LEDs for serial activity, and two green LEDs for system status. The green LEDs will be enabled when the IF power is at a nominal level.","title":"Physical Interface"},{"location":"hardware/fpga/","text":"Digital Backend TODO! Describe hardware interfaces of SNAP.","title":"Digital Backend"},{"location":"hardware/fpga/#digital-backend","text":"TODO! Describe hardware interfaces of SNAP.","title":"Digital Backend"},{"location":"hardware/overview/","text":"Hardware Overview The GReX hardware system has several \"top level\" components, which constitute the entire system. These include the feed antenna and low noise amplifiers (LNA), the frontend module , the digital backend , and of course the server. The following diagrams lay out general overview of the interconnections. Showing them all at once would be a bit much, so they're broken down here into discrete kinds of signals. RF Signal Path flowchart TD A[Feed] B[FEM] C[SNAP] D[Server] A --> L1[LNA] A --> L2[LNA] L1 -->|H Pol| B L2 -->|V Pol| B subgraph The Box B -->|H Pol| C B -->|V Pol| C end C -->|10 GbE| D[Server] Power Distribution flowchart BT L1[LNA] L2[LNA] B[FEM] C[SNAP] S[Switching Supply] R[Custom Linear PSU] P[Raspberry Pi] M[Mains Power] V[Synthesizer] G[GPS Receiver] M -->|120-240V AC| S subgraph The Box R -->|6.5V DC| V R -->|12V DC| G S -->|12V DC| C S -->|12V DC| R R -->|6.5V DC| B R -->|5V DC| P end B --->|5.5V DC| L1 B --->|5.5V DC| L2 Clocks, References and Timing flowchart BT B[FEM] V[Synthesizer] G[GPS Receiver] S[SNAP] A[GPS Antenna] subgraph The Box G -->|10 MHz| V G -->|PPS| S V -->|500 MHz| S V -->|1030 MHz| B end A --> G Monitor and Control flowchart TB B[FEM] S[SNAP] P[Raspberry Pi] D[Server] subgraph The Box P <-->|UART| B S <-->|GPIO| P end P <--->|1 GbE| D","title":"Overview"},{"location":"hardware/overview/#hardware-overview","text":"The GReX hardware system has several \"top level\" components, which constitute the entire system. These include the feed antenna and low noise amplifiers (LNA), the frontend module , the digital backend , and of course the server. The following diagrams lay out general overview of the interconnections. Showing them all at once would be a bit much, so they're broken down here into discrete kinds of signals.","title":"Hardware Overview"},{"location":"hardware/overview/#rf-signal-path","text":"flowchart TD A[Feed] B[FEM] C[SNAP] D[Server] A --> L1[LNA] A --> L2[LNA] L1 -->|H Pol| B L2 -->|V Pol| B subgraph The Box B -->|H Pol| C B -->|V Pol| C end C -->|10 GbE| D[Server]","title":"RF Signal Path"},{"location":"hardware/overview/#power-distribution","text":"flowchart BT L1[LNA] L2[LNA] B[FEM] C[SNAP] S[Switching Supply] R[Custom Linear PSU] P[Raspberry Pi] M[Mains Power] V[Synthesizer] G[GPS Receiver] M -->|120-240V AC| S subgraph The Box R -->|6.5V DC| V R -->|12V DC| G S -->|12V DC| C S -->|12V DC| R R -->|6.5V DC| B R -->|5V DC| P end B --->|5.5V DC| L1 B --->|5.5V DC| L2","title":"Power Distribution"},{"location":"hardware/overview/#clocks-references-and-timing","text":"flowchart BT B[FEM] V[Synthesizer] G[GPS Receiver] S[SNAP] A[GPS Antenna] subgraph The Box G -->|10 MHz| V G -->|PPS| S V -->|500 MHz| S V -->|1030 MHz| B end A --> G","title":"Clocks, References and Timing"},{"location":"hardware/overview/#monitor-and-control","text":"flowchart TB B[FEM] S[SNAP] P[Raspberry Pi] D[Server] subgraph The Box P <-->|UART| B S <-->|GPIO| P end P <--->|1 GbE| D","title":"Monitor and Control"},{"location":"software/box/","text":"Box Software There are four devices that require some software configuration: the Valon synthesizer, the SNAP FPGA, the 10 GbE switch, and the Raspberry Pi. These configuration steps should already be performed before we ship a box, but for completeness, here are the steps that we performed. Valon We need to configure the valon synthesizer to act as the LO for the downconverter and the reference clock for the SNAP ADC. Use the GUI tool here to load this configuration file. Next, go to synthesizer -> write registers. Then, save the configuration to flash to preserve this configuration across reboots. Switch With the box connected and powered on, create an SSH relay to the switch's configuration interface with ssh -L 8291 :192.168.88.1:8291 user@ Then, using winbox connect to localhost, select files on the left, and upload this config file . This should trigger a reboot. Raspberry Pi We prepared the RPi image using the standard raspbian lite OS . As part of the initial image creation, we set the hostname to grex-pi and enabled password-based SSH. Using raspi-config , we did the following: - disabled the serial login shell - enabled the hardware serial interface Then, we disabled the hardware's radios by modifying the config.txt file like so . Then, we configured the Pi to have the static IP address of 192.168.0.2 by following this Then, we disabled HCI UART by running sudo systemctl disable hciuart SNAP See SNAP Setup","title":"Box Setup"},{"location":"software/box/#box-software","text":"There are four devices that require some software configuration: the Valon synthesizer, the SNAP FPGA, the 10 GbE switch, and the Raspberry Pi. These configuration steps should already be performed before we ship a box, but for completeness, here are the steps that we performed.","title":"Box Software"},{"location":"software/box/#valon","text":"We need to configure the valon synthesizer to act as the LO for the downconverter and the reference clock for the SNAP ADC. Use the GUI tool here to load this configuration file. Next, go to synthesizer -> write registers. Then, save the configuration to flash to preserve this configuration across reboots.","title":"Valon"},{"location":"software/box/#switch","text":"With the box connected and powered on, create an SSH relay to the switch's configuration interface with ssh -L 8291 :192.168.88.1:8291 user@ Then, using winbox connect to localhost, select files on the left, and upload this config file . This should trigger a reboot.","title":"Switch"},{"location":"software/box/#raspberry-pi","text":"We prepared the RPi image using the standard raspbian lite OS . As part of the initial image creation, we set the hostname to grex-pi and enabled password-based SSH. Using raspi-config , we did the following: - disabled the serial login shell - enabled the hardware serial interface Then, we disabled the hardware's radios by modifying the config.txt file like so . Then, we configured the Pi to have the static IP address of 192.168.0.2 by following this Then, we disabled HCI UART by running sudo systemctl disable hciuart","title":"Raspberry Pi"},{"location":"software/box/#snap","text":"See SNAP Setup","title":"SNAP"},{"location":"software/data_processing/","text":"Data Processing Voltage Dumps The voltage dumps are stored in NetCDF , a machine-independent data format intended for array-oriented scientific data such as ours. This format is a subset of HDF5 and is supported by all major programming languages. Not only that, but it is self describing! Our files contain axis variables on each dimension to remove any ambiguity as to which coordinate each data point refers to. Using the NetCDF binaries, we can quickly introspect a voltage dump to see how it is laid out user@grex:/hdd/data/voltages$ ncdump -h grex_dump-20240319T182522.nc netcdf grex_dump-20240319T182522 { dimensions: time = 1048576 ; pol = 2 ; freq = 2048 ; reim = 2 ; variables: double time ( time ) ; time:units = \"Days\" ; time:long_name = \"TAI days since the MJD Epoch\" ; string pol ( pol ) ; pol:long_name = \"Polarization\" ; double freq ( freq ) ; freq:units = \"Megahertz\" ; freq:long_name = \"Frequency\" ; string reim ( reim ) ; reim:long_name = \"Complex\" ; byte voltages ( time, pol, freq, reim ) ; voltages:long_name = \"Channelized Voltages\" ; voltages:units = \"Volts\" ; } Part of the trickiness here is HDF5 (and NetCDF by extension) doesn't support complex numbers, so the real and imaginary components are stored independently as another dimension. Python Reading Example In Python, an excellent library for dealing with dimensional data is xarray . This library supports a large number of file formats, which are optional dependencies. For us, we need the netCDF4 python package, but it's probably best just to use the complete installation. It might also help if you read the xarray page on NetCDF . Reading this file with xarray . open_dataset ( \"grex_dump.nc\" ) will associate all the axes to the appropriate dimensions, we just have to construct the complex numbers manually. A one-liner to do this would be voltages = ds [ \"voltages\" ] . sel ( reim = \"real\" ) + ds [ \"voltages\" ] . sel ( reim = \"imaginary\" ) * 1 j This may take a while for large dumps though, and you may want to slice it up into chunks (not sure if there is an elegant way to do that here). But now you can do eveything xarray can do! A few examples: Stokes I Calculated by taking the time-average of the sum of magnitude squared of the voltages stokesi = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = 'pol' , dtype = 'int32' ) . sum ( dim = 'reim' , dtype = 'int32' ) Note that the original voltages are in \"int8\" format which is not enough to store squared voltages, and thus we are converting to \"int32\". H1 Line Get the Stokes intensity around the H1 line (+/- 1 MHz) as a function a time h1 = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = \"pol\" , dtype = 'int32' ) . sel ( freq = slice ( 1421 , 1419 )) . sum ( dim = \"freq\" , dtype = 'int32' )","title":"Data Processing"},{"location":"software/data_processing/#data-processing","text":"","title":"Data Processing"},{"location":"software/data_processing/#voltage-dumps","text":"The voltage dumps are stored in NetCDF , a machine-independent data format intended for array-oriented scientific data such as ours. This format is a subset of HDF5 and is supported by all major programming languages. Not only that, but it is self describing! Our files contain axis variables on each dimension to remove any ambiguity as to which coordinate each data point refers to. Using the NetCDF binaries, we can quickly introspect a voltage dump to see how it is laid out user@grex:/hdd/data/voltages$ ncdump -h grex_dump-20240319T182522.nc netcdf grex_dump-20240319T182522 { dimensions: time = 1048576 ; pol = 2 ; freq = 2048 ; reim = 2 ; variables: double time ( time ) ; time:units = \"Days\" ; time:long_name = \"TAI days since the MJD Epoch\" ; string pol ( pol ) ; pol:long_name = \"Polarization\" ; double freq ( freq ) ; freq:units = \"Megahertz\" ; freq:long_name = \"Frequency\" ; string reim ( reim ) ; reim:long_name = \"Complex\" ; byte voltages ( time, pol, freq, reim ) ; voltages:long_name = \"Channelized Voltages\" ; voltages:units = \"Volts\" ; } Part of the trickiness here is HDF5 (and NetCDF by extension) doesn't support complex numbers, so the real and imaginary components are stored independently as another dimension.","title":"Voltage Dumps"},{"location":"software/data_processing/#python-reading-example","text":"In Python, an excellent library for dealing with dimensional data is xarray . This library supports a large number of file formats, which are optional dependencies. For us, we need the netCDF4 python package, but it's probably best just to use the complete installation. It might also help if you read the xarray page on NetCDF . Reading this file with xarray . open_dataset ( \"grex_dump.nc\" ) will associate all the axes to the appropriate dimensions, we just have to construct the complex numbers manually. A one-liner to do this would be voltages = ds [ \"voltages\" ] . sel ( reim = \"real\" ) + ds [ \"voltages\" ] . sel ( reim = \"imaginary\" ) * 1 j This may take a while for large dumps though, and you may want to slice it up into chunks (not sure if there is an elegant way to do that here). But now you can do eveything xarray can do! A few examples:","title":"Python Reading Example"},{"location":"software/data_processing/#stokes-i","text":"Calculated by taking the time-average of the sum of magnitude squared of the voltages stokesi = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = 'pol' , dtype = 'int32' ) . sum ( dim = 'reim' , dtype = 'int32' ) Note that the original voltages are in \"int8\" format which is not enough to store squared voltages, and thus we are converting to \"int32\".","title":"Stokes I"},{"location":"software/data_processing/#h1-line","text":"Get the Stokes intensity around the H1 line (+/- 1 MHz) as a function a time h1 = np . square ( abs ( voltages ), dtype = 'int32' ) . sum ( dim = \"pol\" , dtype = 'int32' ) . sel ( freq = slice ( 1421 , 1419 )) . sum ( dim = \"freq\" , dtype = 'int32' )","title":"H1 Line"},{"location":"software/operation/","text":"Operation We'll flesh out all the details on operation soon, but in the meantime here are the critical details. Turning on the SNAP To turn on the SNAP, SSH into the Pi (as discussed in the server setup), Then on the pi create (if it doesn't already exist) a bash script called snap.sh with the following: Warning This is assuming you have a V2 power supply, ON and OFF are reversed otherwise #!/bin/env bash # Usage: ./snap.sh BASE_GPIO_PATH = /sys/class/gpio PWN_PIN = 20 if [ ! -e $BASE_GPIO_PATH /gpio $PWN_PIN ] ; then echo \"20\" > $BASE_GPIO_PATH /export fi echo \"out\" > $BASE_GPIO_PATH /gpio $PWN_PIN /direction if [[ -z $1 ]] ; then echo \"Please pass `on` or `off` as an argument\" else case $1 in \"on\" | \"ON\" ) echo \"0\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; \"off\" | \"OFF\" ) echo \"1\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; * ) echo \"Please pass `on` or `off` as an argument\" exit -1 ;; esac fi exit 0 Make it executable is chmod +x snap.sh , and use ./snap.sh to control the power state of the SNAP. If you get permission errors when doing so, make sure your Pi's user is a member of the gpio group. You can add your user to the group with: sudo usermod -a -G gpio $( whoami ) then relog and try again. Running the Pipeline In the grex folder, under pipeline there is the single bash script that runs the pipeline. Simply calling it ./grex.sh should start everything up. By default, it will run the normal detection pipeline. If you want to just run T0 (packet capture and first stage processing), remove the final line that calls psrdada and replace it (the whole line) with filterbank . Triggering voltage dumps Normally, T2 will send triggers to T0 to dump the voltage ringbuffer to disk. You can emulate this by sending a UDP packet to the trigger socket any other way. One simple way is with bash echo \" \" > /dev/udp/localhost/65432 SSH Port Tunneling In some circumstances, it may be useful to access ports on the GReX server on your local computer remotely. We can accomplish this using SSH Tunneling . One example of why we might want to do this is to access the 10 GbE switch configuration that is located in the far-side GReX box. It runs a normal web page on a static ip of 192.168.88.1 . You can access this from a web browser if you are sitting at the GReX server, but not remotely. To access it using SSH tunneling, we can forward that IP's port 80 (standard HTTP) to our local computer at some unused, non-privaleged port. ssh -L 8080 :192.168.88.1:80 username@grex-server-address Another example is perhaps you want to run a Jupypter Hub instance on the GReX server. In that case, the website it is hosting is on the server itself, so you would run: ssh -L 8080 :localhost:80 username@grex-server-address Another useful one is access to the Prometheus time-series database used for monitoring. That is active on port 9090 ssh -L 9090 :localhost:9090 username@grex-server-address Pulse Injection If you need to generate and inject fake pulses into raw voltages to test the pipeline, Liam Connor's injection codes contain all the relevant tools. In a Python Jupyter notebook, you can import this Python script and use the functions within to generate a fake pulse and write it to an output .dat file. import simulate_frb dt = 8.192e-6 width_sec = 2 * dt # depending on your preferred pulse width Nfreq = 2048 data , params = simulate_frb . gen_simulated_frb ( NFREQ = Nfreq , NTIME = 16384 , sim = True , fluence = 70 , # This controls the pulse SNR spec_ind = 0.0 , width = width_sec , dm = 100. , # dispersion measure background_noise = np . zeros ([ 2048 , 16384 ]), delta_t = dt , plot_burst = False , freq = ( 1530.0 , 1280.0 ), FREQ_REF = 1405.0 , scintillate = False , scat_tau_ref = 0.0 , disp_ind = 2.0 , conv_dmsmear = False , ) Before writing to an output file, you might want to check what the pulse looks like and its histogram distribution, ensuring it\u2019s not all zeros due to conversion to int8 (bit type of raw voltages). from matplotlib import pyplot as plt plt . hist ( data . astype ( np . int8 ) . flatten (), bins = 50 ) plt . semilogy () plt . show () ### also, visualize the pulse itself plt . imshow ( data . astype ( np . int8 ), aspect = 'auto' ) plt . colorbar () plt . show () If the pulse looks reasonable, we can convert to int8 type and write to an output .dat file. with open ( 'test_injpulse0.dat' , 'wb' ) as f : f . write ( data . astype ( np . int8 ) . tobytes ()) Move this .dat file to ~/grex/pipeline/fake/ , the specified directory in grex.sh , to actually inject the pulse. Then you could change the injection cadence (in seconds) in ~/grex/pipeline/.env by adding the following line. injection_cadence = 300 Now you are good to go. Querying Historical Data As part of the server setup, we've hooked up a time-series database (Prometheus) to T0, which is used to monitor the state of the telescope. This includes recording timestamped values for temperature, ADC counts, and integrated Stokes I data. It may be useful to query this data, which is relativly easy to do programatically using Prometheus' HTTP API . You can explore the database by accessing http://localhost:9090 on the server (or with SSH-forwarding described above). H1 Example Say you want to query historical Stokes I data around the H1 frequency of 1420 MHz. This data is stored in \"channels\" where the index is the FPGA's channel number. If you recall, we operate in the first Nyquist zone, so the spectrum is flipped, where channel 0 represents 1530 MHz and channel 2048 represents 1280 MHz. Using the requests library in Python, we can perform the HTTP request to the database. import requests import time import numpy as np # Get the current (UNIX) time now = time . time () # We want data, say two weeks back past = now - 60 * 60 * 24 * 7 * 2 # We want data around 1420 MHz, so around channel 893 # Might as well grab the block of data around it, so we can watch it change in frequency with time channels = [ 889 , 890 , 891 , 892 , 893 , 894 , 895 ] data = [] uri_base = 'http://127.0.0.1:9090/api/v1/query_range' for channel in channels : params = { \"query\" : f 'spectrum {{ channel=\" { channel } \" }} ' , \"start\" : then , \"end\" : now , \"step\" : \"10m\" } # Depending on the timespan, there's a maximum size per query resp = requests . get ( uri_base , params = params ) # Extract the data and convert to floats vals = resp . json ()[ 'data' ][ 'result' ][ 0 ][ 'values' ] data . append ( np . array ( vals ) . astype ( float )) # Restructure the data into a tensor data = np . stack ( data ) # The time axis is duplicated many times, we can extract it by looking at one chunk timestamps = data [ 0 ,:, 0 ] # And then finally extract our 2D block of time/freq linear power data # Indexed in [channel, time] spectra = data [:,:, 1 ]","title":"Operation"},{"location":"software/operation/#operation","text":"We'll flesh out all the details on operation soon, but in the meantime here are the critical details.","title":"Operation"},{"location":"software/operation/#turning-on-the-snap","text":"To turn on the SNAP, SSH into the Pi (as discussed in the server setup), Then on the pi create (if it doesn't already exist) a bash script called snap.sh with the following: Warning This is assuming you have a V2 power supply, ON and OFF are reversed otherwise #!/bin/env bash # Usage: ./snap.sh BASE_GPIO_PATH = /sys/class/gpio PWN_PIN = 20 if [ ! -e $BASE_GPIO_PATH /gpio $PWN_PIN ] ; then echo \"20\" > $BASE_GPIO_PATH /export fi echo \"out\" > $BASE_GPIO_PATH /gpio $PWN_PIN /direction if [[ -z $1 ]] ; then echo \"Please pass `on` or `off` as an argument\" else case $1 in \"on\" | \"ON\" ) echo \"0\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; \"off\" | \"OFF\" ) echo \"1\" > $BASE_GPIO_PATH /gpio $PWN_PIN /value ;; * ) echo \"Please pass `on` or `off` as an argument\" exit -1 ;; esac fi exit 0 Make it executable is chmod +x snap.sh , and use ./snap.sh to control the power state of the SNAP. If you get permission errors when doing so, make sure your Pi's user is a member of the gpio group. You can add your user to the group with: sudo usermod -a -G gpio $( whoami ) then relog and try again.","title":"Turning on the SNAP"},{"location":"software/operation/#running-the-pipeline","text":"In the grex folder, under pipeline there is the single bash script that runs the pipeline. Simply calling it ./grex.sh should start everything up. By default, it will run the normal detection pipeline. If you want to just run T0 (packet capture and first stage processing), remove the final line that calls psrdada and replace it (the whole line) with filterbank .","title":"Running the Pipeline"},{"location":"software/operation/#triggering-voltage-dumps","text":"Normally, T2 will send triggers to T0 to dump the voltage ringbuffer to disk. You can emulate this by sending a UDP packet to the trigger socket any other way. One simple way is with bash echo \" \" > /dev/udp/localhost/65432","title":"Triggering voltage dumps"},{"location":"software/operation/#ssh-port-tunneling","text":"In some circumstances, it may be useful to access ports on the GReX server on your local computer remotely. We can accomplish this using SSH Tunneling . One example of why we might want to do this is to access the 10 GbE switch configuration that is located in the far-side GReX box. It runs a normal web page on a static ip of 192.168.88.1 . You can access this from a web browser if you are sitting at the GReX server, but not remotely. To access it using SSH tunneling, we can forward that IP's port 80 (standard HTTP) to our local computer at some unused, non-privaleged port. ssh -L 8080 :192.168.88.1:80 username@grex-server-address Another example is perhaps you want to run a Jupypter Hub instance on the GReX server. In that case, the website it is hosting is on the server itself, so you would run: ssh -L 8080 :localhost:80 username@grex-server-address Another useful one is access to the Prometheus time-series database used for monitoring. That is active on port 9090 ssh -L 9090 :localhost:9090 username@grex-server-address","title":"SSH Port Tunneling"},{"location":"software/operation/#pulse-injection","text":"If you need to generate and inject fake pulses into raw voltages to test the pipeline, Liam Connor's injection codes contain all the relevant tools. In a Python Jupyter notebook, you can import this Python script and use the functions within to generate a fake pulse and write it to an output .dat file. import simulate_frb dt = 8.192e-6 width_sec = 2 * dt # depending on your preferred pulse width Nfreq = 2048 data , params = simulate_frb . gen_simulated_frb ( NFREQ = Nfreq , NTIME = 16384 , sim = True , fluence = 70 , # This controls the pulse SNR spec_ind = 0.0 , width = width_sec , dm = 100. , # dispersion measure background_noise = np . zeros ([ 2048 , 16384 ]), delta_t = dt , plot_burst = False , freq = ( 1530.0 , 1280.0 ), FREQ_REF = 1405.0 , scintillate = False , scat_tau_ref = 0.0 , disp_ind = 2.0 , conv_dmsmear = False , ) Before writing to an output file, you might want to check what the pulse looks like and its histogram distribution, ensuring it\u2019s not all zeros due to conversion to int8 (bit type of raw voltages). from matplotlib import pyplot as plt plt . hist ( data . astype ( np . int8 ) . flatten (), bins = 50 ) plt . semilogy () plt . show () ### also, visualize the pulse itself plt . imshow ( data . astype ( np . int8 ), aspect = 'auto' ) plt . colorbar () plt . show () If the pulse looks reasonable, we can convert to int8 type and write to an output .dat file. with open ( 'test_injpulse0.dat' , 'wb' ) as f : f . write ( data . astype ( np . int8 ) . tobytes ()) Move this .dat file to ~/grex/pipeline/fake/ , the specified directory in grex.sh , to actually inject the pulse. Then you could change the injection cadence (in seconds) in ~/grex/pipeline/.env by adding the following line. injection_cadence = 300 Now you are good to go.","title":"Pulse Injection"},{"location":"software/operation/#querying-historical-data","text":"As part of the server setup, we've hooked up a time-series database (Prometheus) to T0, which is used to monitor the state of the telescope. This includes recording timestamped values for temperature, ADC counts, and integrated Stokes I data. It may be useful to query this data, which is relativly easy to do programatically using Prometheus' HTTP API . You can explore the database by accessing http://localhost:9090 on the server (or with SSH-forwarding described above).","title":"Querying Historical Data"},{"location":"software/operation/#h1-example","text":"Say you want to query historical Stokes I data around the H1 frequency of 1420 MHz. This data is stored in \"channels\" where the index is the FPGA's channel number. If you recall, we operate in the first Nyquist zone, so the spectrum is flipped, where channel 0 represents 1530 MHz and channel 2048 represents 1280 MHz. Using the requests library in Python, we can perform the HTTP request to the database. import requests import time import numpy as np # Get the current (UNIX) time now = time . time () # We want data, say two weeks back past = now - 60 * 60 * 24 * 7 * 2 # We want data around 1420 MHz, so around channel 893 # Might as well grab the block of data around it, so we can watch it change in frequency with time channels = [ 889 , 890 , 891 , 892 , 893 , 894 , 895 ] data = [] uri_base = 'http://127.0.0.1:9090/api/v1/query_range' for channel in channels : params = { \"query\" : f 'spectrum {{ channel=\" { channel } \" }} ' , \"start\" : then , \"end\" : now , \"step\" : \"10m\" } # Depending on the timespan, there's a maximum size per query resp = requests . get ( uri_base , params = params ) # Extract the data and convert to floats vals = resp . json ()[ 'data' ][ 'result' ][ 0 ][ 'values' ] data . append ( np . array ( vals ) . astype ( float )) # Restructure the data into a tensor data = np . stack ( data ) # The time axis is duplicated many times, we can extract it by looking at one chunk timestamps = data [ 0 ,:, 0 ] # And then finally extract our 2D block of time/freq linear power data # Indexed in [channel, time] spectra = data [:,:, 1 ]","title":"H1 Example"},{"location":"software/overview/","text":"Software Overview There are two primary components to the software stack in GReX. First, the SNAP board must be configured and setup to send voltage data to the server. After that, the pipeline software should take care of the rest. Pipeline Overview flowchart TD A[SNAP] -->|UDP| B[T0] B -->|PSRDADA| C[RFI Cleaning] C -->|PSRDADA| D[Heimdall] D -->|Sockets| E[T2] Software Manifesto To limit downtime and maximize reproducability, we will try to adopt a consistent software development strategy. Primarily: Builds will be deterministic and reproducible Code will be version controlled, organized, and public Language Specific Rust No clippy warnings Avoid unsafe Document everything Formatted with rustfmt C++ Code will be formatted with clang-format's LLVM style Try to minimize (solve) all errors from -Wall Python Code will be formatted with Black Docstrings will follow the numpy format Gradual typing will be used (PEP 438) and checked with mypy or equivalent Environments will have pinned dependencies (reproducible) Most of this can be accomplised by using Poetry .","title":"Overview"},{"location":"software/overview/#software-overview","text":"There are two primary components to the software stack in GReX. First, the SNAP board must be configured and setup to send voltage data to the server. After that, the pipeline software should take care of the rest.","title":"Software Overview"},{"location":"software/overview/#pipeline-overview","text":"flowchart TD A[SNAP] -->|UDP| B[T0] B -->|PSRDADA| C[RFI Cleaning] C -->|PSRDADA| D[Heimdall] D -->|Sockets| E[T2]","title":"Pipeline Overview"},{"location":"software/overview/#software-manifesto","text":"To limit downtime and maximize reproducability, we will try to adopt a consistent software development strategy. Primarily: Builds will be deterministic and reproducible Code will be version controlled, organized, and public","title":"Software Manifesto"},{"location":"software/overview/#language-specific","text":"","title":"Language Specific"},{"location":"software/overview/#rust","text":"No clippy warnings Avoid unsafe Document everything Formatted with rustfmt","title":"Rust"},{"location":"software/overview/#c","text":"Code will be formatted with clang-format's LLVM style Try to minimize (solve) all errors from -Wall","title":"C++"},{"location":"software/overview/#python","text":"Code will be formatted with Black Docstrings will follow the numpy format Gradual typing will be used (PEP 438) and checked with mypy or equivalent Environments will have pinned dependencies (reproducible) Most of this can be accomplised by using Poetry .","title":"Python"},{"location":"software/pipeline/","text":"Pipeline","title":"Pipeline"},{"location":"software/pipeline/#pipeline","text":"","title":"Pipeline"},{"location":"software/server_setup/","text":"Server Setup Once you get your server (either from Puget systems or otherwise), we need to setup additional hardware, adjust some system settings, setup networking, and install the pipeline software. Hardware Setup The server straight from Puget does not have the GPU or 10 GbE network card installed, we will do this first. Open the case and remove the GPU retention bracket Remove the test GPU (T1000), keep this for later in case we need an aditional video output Install the 3090 Ti in the first GPU slot, this will take up three slots of space Install the network card in the bottom slot Wire the 3090 Ti power cable to the harness provided by Puget (they knew this was the GPU we were going to install) Remove the GPU retention clips from the retention bracket that would interfere with the card. It's too tall for it anyway. Replace the retention bracket and close the case. Finally, we need to hook up a monitor to the 3090 Ti so we can setup the software Initial OS Setup On boot, you will be presented with the Ubuntu graphical interface. If a password is requested, the deafult is provided in the information booklet that came with the hardware. First, we will change the system password. Set this to something memorable. passwd $( whoami ) Now, we will update the system. There shouldn't be many updates if this is a new machine. sudo apt-get update sudo apt-get upgrade -y We will be editing a bunch of files, if you are comfy in the command line, you probably want to install some editors. Otherwise the graphical gedit tool will be fine. sudo apt-get install emacs vim -y Finally, we will set the hostname. We'll be using the grex-- paradigm (just for clarity, no real reason not to). As in, the first server that is managed by Caltech at OVRO will be grex-caltech-ovro . sudo hostnamectl set-hostname Some updates may require a reboot. If it asks, do that now. Networking Now, we need to setup the networking for the GReX system. We will operate under the assumption that the internet-facing connection will get an IP address from a DHCP server. If that is not the case, consult whoever runs your network on the appropriate setup. Regardless of the WAN connection, the 10 GbE fiber connection to the GReX terminal will be configured the same. Overview The 10 GbE fiber port serves a few purposes. It is the main data transfer link between the FPGA in the field and the server, but it also carries the monitor and control for the box. This monitor and control connection includes the SNAP control connection and the Raspberry Pi. The SNAP requires an external DHCP server, which we have to provide on this port. Additionally, the 10 GbE switch in the box has its own default subnet for configuration ( 192.168.88.X ). To make everything talk to each other, we need to add two IPs on this port: one in the subnet of the switch's config interface, and the other for DHCP of the various devices. Netplan In /etc/netplan remove any files that are currently there. Check whether you are using NetworkManager or networkd: systemctl status NetworkManager systemctl status systemd-networkd If NetworkManager is running and networkd is not, disable NetworkManager and enable networkd. (Otherwise, skip this step.) sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo systemctl enable systemd-networkd Then, create a new file called config.yaml with the following contents network : version : 2 renderer : networkd ethernets : # Two WAN interfaces. Configure this according to your network setup enp36s0f0 : dhcp4 : true enp36s0f1 : dhcp4 : true # 10 GbE connection over fiber to the box enp1s0f0 : mtu : 9000 addresses : - 192.168.0.1/24 - 192.168.88.2/24 Then apply with sudo netplan apply DHCP Server Now, we need to setup the DHCP server on the 10 GbE port. First, we install the DHCP server software: sudo apt-get install dnsmasq Create the configuration file in /etc/dnsmasq.conf # Only bind to the 10 GbE interface interface = enp1s0f0 # Disable DNS port = 0 # DHCP Options dhcp-range = 192.168.0.0,static dhcp-option = option:router,192.168.0.1 dhcp-option = option:netmask,255.255.255.0 #dhcp-host=,192.168.0.3,snap log-async log-queries log-dhcp Then, enable the DHCP server service sudo systemctl enable dnsmasq --now This sets up a very simple DHCP server that will give the IP address 192.168.0.3 to the SNAP. Unfortunately, the folks who set up the networking interface for the SNAP only provide a DHCP interface and a dynamic (non-observable) MAC address (for some reason). As such, we have to now turn on the SNAP, wait for it to try to get an IP address from dnsmasq so we know it's MAC, then update the dhcp-host line and restart the DHCP server. Power cycle the SNAP (or turn it on if it wasn't turned on yet) following the instructions in operation Wait a moment and open the log of dnsmasq with journalctl -u dnsmasq , skip to the bottom with G (Shift + g) You should see a line like Aug 16 14:39:06 grex-caltech-ovro dnsmasq-dhcp[5115]: 1085377743 DHCPDISCOVER(enp1s0f0) 00:40:bf:06:13:02 no address available This implies the SNAP has a MAC address of 00:40:bf:06:13:02 (yours will be different). 4. Go back and uncomment and edit the dhcp-host line of /etc/dnsmasq.conf to contain this MAC. For example, in this case we would put dhcp-host=00:40:bf:06:13:02,192.168.0.3,snap 5. Finally, restart the dhcp server with sudo systemctl restart dnsmasq After waiting a bit for the SNAP to send a new request for a DHCP lease, look at the latest logs again from journalctl. If it ends with something like Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPREQUEST(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 tags: known, enp1s0f0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPACK(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 snap Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 requested options: 1:netmask, 3:router, 28:broadcast, 6:dns-server Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 next server: 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 1 option: 53 message-type 5 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 54 server-identifier 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 51 lease-time 1h Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 58 T1 30m Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 59 T2 52m30s Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 28 broadcast 192.168.0.255 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 1 netmask 255.255.255.0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 3 router 192.168.0.1 That means the SNAP got an IP. You should now be able to ping 192.168.0.3 to make sure it's alive. Advanced 10 GbE Settings Unfortunately, the OS's default configuration for the 10 GbE network card is not optimized for our use-case of streaming time domain science data. As such, we need to adjust a few things. Create the file /etc/sysctl.d/20-grex.conf with the following contents: kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.core.optmem_max = 16777216 vm.swappiness=1 Then apply these changes with sudo sysctl --system Now, we need a program called ethtool to apply some more settings sudo apt-get install ethtool -y Now we will create a file to run on boot to apply a sequence of ethtool settings. Create the file /etc/rc.local with the following contents: #!/bin/env bash ethtool -G enp1s0f0 rx 4096 tx 4096 ethtool -A enp1s0f0 rx on ethtool -A enp1s0f0 tx on Make this file executable with sudo chmod +x /etc/rc.local Now create the file /etc/systemd/system/rc-local.service with the following contents: [Unit] Description = /etc/rc.local Compatibility ConditionPathExists = /etc/rc.local [Service] Type = forking ExecStart = /etc/rc.local start TimeoutSec = 0 StandardOutput = tty RemainAfterExit = yes SysVStartPriority = 99 [Install] WantedBy = multi-user.target Then enable the rc-local service sudo systemctl enable rc-local Finally, reboot GPU Drivers / CUDA Heimdall (the pulse detection part of our pipeline) relies on the CUDA toolkit. Let's install that now (version 12.3) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install cuda-toolkit-12-3 And then install the open-source kernel drivers (Version 545) (overwriting previously installed ones) sudo apt-get install -y nvidia-kernel-open-545 sudo apt-get install -y cuda-drivers-545 You may want to run the following to cleanup old dependencies/driver versions that may have been preinstalled sudo apt-get autoremove Reboot. Then run nvidia-smi to see if the CUDA version and Driver version came up correctly. Finally, add the following to ~/.bashrc to let our use use CUDA # CUDA 12.3 export PATH = /usr/local/cuda-12.3/bin ${ PATH :+: ${ PATH }} export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/cuda-12.3/lib64 and source ~/.bashrc or relog. Warning If you change CUDA versions, you'll need to update these paths! Pipeline Dependencies PSRDADA We use PSRDADA to connect the packet capture and first stage processing pipeline T0 to the pulse detection framework heimdall . This is a library we need to install. We will build a few programs, so might as well create a directory to do this in to keep our home directory organized. mkdir src && cd src Then clone PSRDADA git clone git://git.code.sf.net/p/psrdada/code psrdada && cd psrdada # Last tested version, bump as appropriate git checkout 008afa7 Now, install some build dependencies sudo apt-get install build-essential cmake ninja-build -y Then, build PSRDADA mkdir build && cd build cmake -GNinja .. ninja sudo ninja install This will install the control programs and libraries to /usr/local/bin and /usr/local/lib , respectively. We have to add the latter to out linker path, by adding the following to ~./bashrc # PSRDADA export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/lib Then, relog once agian. Heimdall Similar process to build the pulse-detection software, heimdall. First, clone our fork in our ~/src directory: git clone --recurse-submodules https://github.com/GReX-Telescope/heimdall-astro cd heimdall-astro Install some build dependencies sudo apt-get install libboost-all-dev -y Then build mkdir build && cd build cmake -GNinja .. ninja Run the test dedispersion program to make sure everything seemed to work ./dedisp/testdedisp which should return ----------------------------- INPUT DATA --------------------------------- Frequency of highest chanel (MHz) : 1581.0000 Bandwidth (MHz) : 100.00 NCHANS (Channel Width [MHz]) : 1024 (-0.097656) Sample time (after downsampling by 1) : 0.000250 Observation duration (s) : 30.000000 (119999 samples) Data RMS ( 8 bit input data) : 25.000000 Input data array size : 468 MB Embedding signal ----------------------------- INJECTED SIGNAL ---------------------------- Pulse time at f0 (s) : 3.141590 (sample 12566) Pulse DM (pc/cm^3) : 41.159000 Signal Delays : 0.000000, 0.000008, 0.000017 ... 0.009530 Rawdata Mean (includes signal) : -0.002202 Rawdata StdDev (includes signal) : 25.001451 Pulse S/N (per frequency channel) : 1.000000 Quantizing array Quantized data Mean (includes signal) : 127.497818 Quantized data StdDev (includes signal) : 25.003092 Init GPU Create plan Gen DM list ----------------------------- DM COMPUTATIONS ---------------------------- Computing 32 DMs from 2.000000 to 102.661667 pc/cm^3 Max DM delay is 95 samples (0 seconds) Computing 119904 out of 119999 total samples (99.92% efficiency) Output data array size : 14 MB Compute on GPU Dedispersion took 0.02 seconds Output RMS : 0.376464 Output StdDev : 0.002307 DM trial 11 (37.681 pc/cm^3), Samp 12566 (3.141500 s): 0.390678 (6.16 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12567 (3.141750 s): 0.398160 (9.41 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12568 (3.142000 s): 0.393198 (7.25 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12569 (3.142250 s): 0.391713 (6.61 sigma) DM trial 12 (40.926 pc/cm^3), Samp 12566 (3.141500 s): 0.441719 (28.29 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12564 (3.141000 s): 0.400574 (10.45 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12565 (3.141250 s): 0.403097 (11.55 sigma) Dedispersion successful. Finally, install this into our path with sudo ninja install The heimdall executable should now be available for the pipeline as well as offline analysis. HDF5/NetCDF We use the fantastic netcdf library to write out voltage data. To do this, we need to statically link netcdf (and it's dependent HDF5) at compile time. As such, we have to install these libraries systemw-wide. sudo apt-get install libhdf5-dev libnetcdf-dev -y Rust Many parts of the pipeline software are written in the Rust programming language. We will be building this software from scratch, so we need to install the rust compiler and its tooling. This is easy enough with rustup We need curl for the rustup installer, so sudo apt-get install curl -y Then run the installer, using all the default settings curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh Python To get out of version hell for python stuff, we're using Poetry . To install it, we will: curl -sSL https://install.python-poetry.org | python3 - We need to make a few adjustments to ~/.bashrc to correct the paths and fix a bug. Append the following to the end. export PATH = \"/home/user/.local/bin: $PATH \" # Fix the \"Poetry: Failed to unlock the collection\" issue export PYTHON_KEYRING_BACKEND = keyring.backends.null.Keyring Go ahead and source ~/.bashrc now to get these changes in your shell. Pipeline Software To organize all the software needed for running the whole pipeline, we will grab the metapackage from github and clone somewhere (like the home directory): cd git clone --recurse-submodules https://github.com/GReX-Telescope/grex Then, assuming you followed all the previous steps, build the pipeline software with ./grex/build.sh Lastly, you'll need to install the parallel package sudo apt install parallel -y Databases and Metrics Collection Grafana We use Grafana as the observability frontend. Here, we'll assume you are going to set up a Grafana cloud (free-tier) account. While it is possible to use a self-hosted version, we found this to be the most simple. Steps: 1. Create a grafana account here 2. Pick a stack URL that makes sense 3. Click \"Hosted Telemetry Data\" 4. Click \"Hosted Prometheus metrics\" 5. With the default settings of \"via Grafana Alloy\", scroll down to generate an API token (use any name). 6. Record the url , username , and password of the generated Alloy configuration 7. Click the grafana logo in the top left corner and then hit \"Get started\" again. 8. This time, follow \"Hosted Telemetry Data\" with \"OpenTelemetry (OTLP)\" 9. Again, create an API token, scroll down to the generated Alloy configuration file and find the two sections for \"grafana_cloud\" 10. Record the client.endpoint and username and password 11. Again, return to the home screen with the grafana logo, open the menu and navigate to \"Dashboards\" 12. In the top right, go to \"New\" and then \"Import\" 13. Go to our observability stack and either download or copy the dashboard JSON from here 14. Either load or copy the model and hit \"Load\" and then finish the import. This is our main pipeline visualization. 15. Go back to dashboards, import again, but this type supply 1860 for the dashboard ID. This is the \"Node Exporter\" dashboard that will let you monitor all the metrics about your server such as disk usage, network activity, etc. 16. Select our previously setup prometheus data source 17. Now, continue to the docker setup below: Docker Following docker's documentation , install docker the usual way as that is what orchestrates the databases, log aggregation, and communication to grafana. Observability Stack Somewhere obvious (like the home dir), clone the stack . Copy alloy.env.example to alloy.env and fill out the sections according to your grafana configuration. GRAFANA_OTLP_ENDPOINT is client.endpoint from the OpenTelemetry config GRAFANA_OTLP_USER is username from the OpenTelemetry config GRAFANA_OTLP_PASSWORD is username from the OpenTelemetry config GRAFANA_PROM_ENDPOINT is url from the Prometheus config GRAFANA_PROM_USER is the username from the Promethus config GRAFANA_PROM_PASS is the password from the Prometheus config Following the Grafana example above, a completed alloy.env file will look like this Set docker to run as a systemd service with sudo systemctl enable docker if you didn't already in the installation. Test the stack with sudo docker compose up Navigate to your Grafana Node Exporter Full dashboard (it may take a reselect of Datasource to populate) but you should now see the dashboard with all its data. Ctrl-C out of that and then start up the stack in the background with sudo docker compose up -d Pi SSH The last thing will be to configure easy access to the Raspberry Pi's SSH. We can do that by creating a passwordless SSH key and installing it on the pi. We're going to be behind ssh anyway, and the Pi isn't public-facing, so this is a safe thing to do. On the server, generate a new ssh keypair with ssh-keygen -t rsa Then, install it on the pi with ssh-copy-id pi@192.168.0.2 Finally, create an SSH config that automatically supplies the hostname and user: Create a file on the GReX server in ~/.ssh/config with the contents Host pi Hostname 192.168.0.2 User pi Now you can test the easy connection with ssh pi All done! We should now be ready to run the pipeline!","title":"Server Setup"},{"location":"software/server_setup/#server-setup","text":"Once you get your server (either from Puget systems or otherwise), we need to setup additional hardware, adjust some system settings, setup networking, and install the pipeline software.","title":"Server Setup"},{"location":"software/server_setup/#hardware-setup","text":"The server straight from Puget does not have the GPU or 10 GbE network card installed, we will do this first. Open the case and remove the GPU retention bracket Remove the test GPU (T1000), keep this for later in case we need an aditional video output Install the 3090 Ti in the first GPU slot, this will take up three slots of space Install the network card in the bottom slot Wire the 3090 Ti power cable to the harness provided by Puget (they knew this was the GPU we were going to install) Remove the GPU retention clips from the retention bracket that would interfere with the card. It's too tall for it anyway. Replace the retention bracket and close the case. Finally, we need to hook up a monitor to the 3090 Ti so we can setup the software","title":"Hardware Setup"},{"location":"software/server_setup/#initial-os-setup","text":"On boot, you will be presented with the Ubuntu graphical interface. If a password is requested, the deafult is provided in the information booklet that came with the hardware. First, we will change the system password. Set this to something memorable. passwd $( whoami ) Now, we will update the system. There shouldn't be many updates if this is a new machine. sudo apt-get update sudo apt-get upgrade -y We will be editing a bunch of files, if you are comfy in the command line, you probably want to install some editors. Otherwise the graphical gedit tool will be fine. sudo apt-get install emacs vim -y Finally, we will set the hostname. We'll be using the grex-- paradigm (just for clarity, no real reason not to). As in, the first server that is managed by Caltech at OVRO will be grex-caltech-ovro . sudo hostnamectl set-hostname Some updates may require a reboot. If it asks, do that now.","title":"Initial OS Setup"},{"location":"software/server_setup/#networking","text":"Now, we need to setup the networking for the GReX system. We will operate under the assumption that the internet-facing connection will get an IP address from a DHCP server. If that is not the case, consult whoever runs your network on the appropriate setup. Regardless of the WAN connection, the 10 GbE fiber connection to the GReX terminal will be configured the same.","title":"Networking"},{"location":"software/server_setup/#overview","text":"The 10 GbE fiber port serves a few purposes. It is the main data transfer link between the FPGA in the field and the server, but it also carries the monitor and control for the box. This monitor and control connection includes the SNAP control connection and the Raspberry Pi. The SNAP requires an external DHCP server, which we have to provide on this port. Additionally, the 10 GbE switch in the box has its own default subnet for configuration ( 192.168.88.X ). To make everything talk to each other, we need to add two IPs on this port: one in the subnet of the switch's config interface, and the other for DHCP of the various devices.","title":"Overview"},{"location":"software/server_setup/#netplan","text":"In /etc/netplan remove any files that are currently there. Check whether you are using NetworkManager or networkd: systemctl status NetworkManager systemctl status systemd-networkd If NetworkManager is running and networkd is not, disable NetworkManager and enable networkd. (Otherwise, skip this step.) sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo systemctl enable systemd-networkd Then, create a new file called config.yaml with the following contents network : version : 2 renderer : networkd ethernets : # Two WAN interfaces. Configure this according to your network setup enp36s0f0 : dhcp4 : true enp36s0f1 : dhcp4 : true # 10 GbE connection over fiber to the box enp1s0f0 : mtu : 9000 addresses : - 192.168.0.1/24 - 192.168.88.2/24 Then apply with sudo netplan apply","title":"Netplan"},{"location":"software/server_setup/#dhcp-server","text":"Now, we need to setup the DHCP server on the 10 GbE port. First, we install the DHCP server software: sudo apt-get install dnsmasq Create the configuration file in /etc/dnsmasq.conf # Only bind to the 10 GbE interface interface = enp1s0f0 # Disable DNS port = 0 # DHCP Options dhcp-range = 192.168.0.0,static dhcp-option = option:router,192.168.0.1 dhcp-option = option:netmask,255.255.255.0 #dhcp-host=,192.168.0.3,snap log-async log-queries log-dhcp Then, enable the DHCP server service sudo systemctl enable dnsmasq --now This sets up a very simple DHCP server that will give the IP address 192.168.0.3 to the SNAP. Unfortunately, the folks who set up the networking interface for the SNAP only provide a DHCP interface and a dynamic (non-observable) MAC address (for some reason). As such, we have to now turn on the SNAP, wait for it to try to get an IP address from dnsmasq so we know it's MAC, then update the dhcp-host line and restart the DHCP server. Power cycle the SNAP (or turn it on if it wasn't turned on yet) following the instructions in operation Wait a moment and open the log of dnsmasq with journalctl -u dnsmasq , skip to the bottom with G (Shift + g) You should see a line like Aug 16 14:39:06 grex-caltech-ovro dnsmasq-dhcp[5115]: 1085377743 DHCPDISCOVER(enp1s0f0) 00:40:bf:06:13:02 no address available This implies the SNAP has a MAC address of 00:40:bf:06:13:02 (yours will be different). 4. Go back and uncomment and edit the dhcp-host line of /etc/dnsmasq.conf to contain this MAC. For example, in this case we would put dhcp-host=00:40:bf:06:13:02,192.168.0.3,snap 5. Finally, restart the dhcp server with sudo systemctl restart dnsmasq After waiting a bit for the SNAP to send a new request for a DHCP lease, look at the latest logs again from journalctl. If it ends with something like Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPREQUEST(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 tags: known, enp1s0f0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 DHCPACK(enp1s0f0) 192.168.0.3 00:40:bf:06:13:02 snap Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 requested options: 1:netmask, 3:router, 28:broadcast, 6:dns-server Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 next server: 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 1 option: 53 message-type 5 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 54 server-identifier 192.168.0.1 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 51 lease-time 1h Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 58 T1 30m Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 59 T2 52m30s Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 28 broadcast 192.168.0.255 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 1 netmask 255.255.255.0 Aug 16 14:43:02 grex-caltech-ovro dnsmasq-dhcp[6024]: 1085377743 sent size: 4 option: 3 router 192.168.0.1 That means the SNAP got an IP. You should now be able to ping 192.168.0.3 to make sure it's alive.","title":"DHCP Server"},{"location":"software/server_setup/#advanced-10-gbe-settings","text":"Unfortunately, the OS's default configuration for the 10 GbE network card is not optimized for our use-case of streaming time domain science data. As such, we need to adjust a few things. Create the file /etc/sysctl.d/20-grex.conf with the following contents: kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.core.optmem_max = 16777216 vm.swappiness=1 Then apply these changes with sudo sysctl --system Now, we need a program called ethtool to apply some more settings sudo apt-get install ethtool -y Now we will create a file to run on boot to apply a sequence of ethtool settings. Create the file /etc/rc.local with the following contents: #!/bin/env bash ethtool -G enp1s0f0 rx 4096 tx 4096 ethtool -A enp1s0f0 rx on ethtool -A enp1s0f0 tx on Make this file executable with sudo chmod +x /etc/rc.local Now create the file /etc/systemd/system/rc-local.service with the following contents: [Unit] Description = /etc/rc.local Compatibility ConditionPathExists = /etc/rc.local [Service] Type = forking ExecStart = /etc/rc.local start TimeoutSec = 0 StandardOutput = tty RemainAfterExit = yes SysVStartPriority = 99 [Install] WantedBy = multi-user.target Then enable the rc-local service sudo systemctl enable rc-local Finally, reboot","title":"Advanced 10 GbE Settings"},{"location":"software/server_setup/#gpu-drivers-cuda","text":"Heimdall (the pulse detection part of our pipeline) relies on the CUDA toolkit. Let's install that now (version 12.3) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install cuda-toolkit-12-3 And then install the open-source kernel drivers (Version 545) (overwriting previously installed ones) sudo apt-get install -y nvidia-kernel-open-545 sudo apt-get install -y cuda-drivers-545 You may want to run the following to cleanup old dependencies/driver versions that may have been preinstalled sudo apt-get autoremove Reboot. Then run nvidia-smi to see if the CUDA version and Driver version came up correctly. Finally, add the following to ~/.bashrc to let our use use CUDA # CUDA 12.3 export PATH = /usr/local/cuda-12.3/bin ${ PATH :+: ${ PATH }} export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/cuda-12.3/lib64 and source ~/.bashrc or relog. Warning If you change CUDA versions, you'll need to update these paths!","title":"GPU Drivers / CUDA"},{"location":"software/server_setup/#pipeline-dependencies","text":"","title":"Pipeline Dependencies"},{"location":"software/server_setup/#psrdada","text":"We use PSRDADA to connect the packet capture and first stage processing pipeline T0 to the pulse detection framework heimdall . This is a library we need to install. We will build a few programs, so might as well create a directory to do this in to keep our home directory organized. mkdir src && cd src Then clone PSRDADA git clone git://git.code.sf.net/p/psrdada/code psrdada && cd psrdada # Last tested version, bump as appropriate git checkout 008afa7 Now, install some build dependencies sudo apt-get install build-essential cmake ninja-build -y Then, build PSRDADA mkdir build && cd build cmake -GNinja .. ninja sudo ninja install This will install the control programs and libraries to /usr/local/bin and /usr/local/lib , respectively. We have to add the latter to out linker path, by adding the following to ~./bashrc # PSRDADA export LD_LIBRARY_PATH = $LD_LIBRARY_PATH :/usr/local/lib Then, relog once agian.","title":"PSRDADA"},{"location":"software/server_setup/#heimdall","text":"Similar process to build the pulse-detection software, heimdall. First, clone our fork in our ~/src directory: git clone --recurse-submodules https://github.com/GReX-Telescope/heimdall-astro cd heimdall-astro Install some build dependencies sudo apt-get install libboost-all-dev -y Then build mkdir build && cd build cmake -GNinja .. ninja Run the test dedispersion program to make sure everything seemed to work ./dedisp/testdedisp which should return ----------------------------- INPUT DATA --------------------------------- Frequency of highest chanel (MHz) : 1581.0000 Bandwidth (MHz) : 100.00 NCHANS (Channel Width [MHz]) : 1024 (-0.097656) Sample time (after downsampling by 1) : 0.000250 Observation duration (s) : 30.000000 (119999 samples) Data RMS ( 8 bit input data) : 25.000000 Input data array size : 468 MB Embedding signal ----------------------------- INJECTED SIGNAL ---------------------------- Pulse time at f0 (s) : 3.141590 (sample 12566) Pulse DM (pc/cm^3) : 41.159000 Signal Delays : 0.000000, 0.000008, 0.000017 ... 0.009530 Rawdata Mean (includes signal) : -0.002202 Rawdata StdDev (includes signal) : 25.001451 Pulse S/N (per frequency channel) : 1.000000 Quantizing array Quantized data Mean (includes signal) : 127.497818 Quantized data StdDev (includes signal) : 25.003092 Init GPU Create plan Gen DM list ----------------------------- DM COMPUTATIONS ---------------------------- Computing 32 DMs from 2.000000 to 102.661667 pc/cm^3 Max DM delay is 95 samples (0 seconds) Computing 119904 out of 119999 total samples (99.92% efficiency) Output data array size : 14 MB Compute on GPU Dedispersion took 0.02 seconds Output RMS : 0.376464 Output StdDev : 0.002307 DM trial 11 (37.681 pc/cm^3), Samp 12566 (3.141500 s): 0.390678 (6.16 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12567 (3.141750 s): 0.398160 (9.41 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12568 (3.142000 s): 0.393198 (7.25 sigma) DM trial 11 (37.681 pc/cm^3), Samp 12569 (3.142250 s): 0.391713 (6.61 sigma) DM trial 12 (40.926 pc/cm^3), Samp 12566 (3.141500 s): 0.441719 (28.29 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12564 (3.141000 s): 0.400574 (10.45 sigma) DM trial 13 (44.171 pc/cm^3), Samp 12565 (3.141250 s): 0.403097 (11.55 sigma) Dedispersion successful. Finally, install this into our path with sudo ninja install The heimdall executable should now be available for the pipeline as well as offline analysis.","title":"Heimdall"},{"location":"software/server_setup/#hdf5netcdf","text":"We use the fantastic netcdf library to write out voltage data. To do this, we need to statically link netcdf (and it's dependent HDF5) at compile time. As such, we have to install these libraries systemw-wide. sudo apt-get install libhdf5-dev libnetcdf-dev -y","title":"HDF5/NetCDF"},{"location":"software/server_setup/#rust","text":"Many parts of the pipeline software are written in the Rust programming language. We will be building this software from scratch, so we need to install the rust compiler and its tooling. This is easy enough with rustup We need curl for the rustup installer, so sudo apt-get install curl -y Then run the installer, using all the default settings curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh","title":"Rust"},{"location":"software/server_setup/#python","text":"To get out of version hell for python stuff, we're using Poetry . To install it, we will: curl -sSL https://install.python-poetry.org | python3 - We need to make a few adjustments to ~/.bashrc to correct the paths and fix a bug. Append the following to the end. export PATH = \"/home/user/.local/bin: $PATH \" # Fix the \"Poetry: Failed to unlock the collection\" issue export PYTHON_KEYRING_BACKEND = keyring.backends.null.Keyring Go ahead and source ~/.bashrc now to get these changes in your shell.","title":"Python"},{"location":"software/server_setup/#pipeline-software","text":"To organize all the software needed for running the whole pipeline, we will grab the metapackage from github and clone somewhere (like the home directory): cd git clone --recurse-submodules https://github.com/GReX-Telescope/grex Then, assuming you followed all the previous steps, build the pipeline software with ./grex/build.sh Lastly, you'll need to install the parallel package sudo apt install parallel -y","title":"Pipeline Software"},{"location":"software/server_setup/#databases-and-metrics-collection","text":"","title":"Databases and Metrics Collection"},{"location":"software/server_setup/#grafana","text":"We use Grafana as the observability frontend. Here, we'll assume you are going to set up a Grafana cloud (free-tier) account. While it is possible to use a self-hosted version, we found this to be the most simple. Steps: 1. Create a grafana account here 2. Pick a stack URL that makes sense 3. Click \"Hosted Telemetry Data\" 4. Click \"Hosted Prometheus metrics\" 5. With the default settings of \"via Grafana Alloy\", scroll down to generate an API token (use any name). 6. Record the url , username , and password of the generated Alloy configuration 7. Click the grafana logo in the top left corner and then hit \"Get started\" again. 8. This time, follow \"Hosted Telemetry Data\" with \"OpenTelemetry (OTLP)\" 9. Again, create an API token, scroll down to the generated Alloy configuration file and find the two sections for \"grafana_cloud\" 10. Record the client.endpoint and username and password 11. Again, return to the home screen with the grafana logo, open the menu and navigate to \"Dashboards\" 12. In the top right, go to \"New\" and then \"Import\" 13. Go to our observability stack and either download or copy the dashboard JSON from here 14. Either load or copy the model and hit \"Load\" and then finish the import. This is our main pipeline visualization. 15. Go back to dashboards, import again, but this type supply 1860 for the dashboard ID. This is the \"Node Exporter\" dashboard that will let you monitor all the metrics about your server such as disk usage, network activity, etc. 16. Select our previously setup prometheus data source 17. Now, continue to the docker setup below:","title":"Grafana"},{"location":"software/server_setup/#docker","text":"Following docker's documentation , install docker the usual way as that is what orchestrates the databases, log aggregation, and communication to grafana.","title":"Docker"},{"location":"software/server_setup/#observability-stack","text":"Somewhere obvious (like the home dir), clone the stack . Copy alloy.env.example to alloy.env and fill out the sections according to your grafana configuration. GRAFANA_OTLP_ENDPOINT is client.endpoint from the OpenTelemetry config GRAFANA_OTLP_USER is username from the OpenTelemetry config GRAFANA_OTLP_PASSWORD is username from the OpenTelemetry config GRAFANA_PROM_ENDPOINT is url from the Prometheus config GRAFANA_PROM_USER is the username from the Promethus config GRAFANA_PROM_PASS is the password from the Prometheus config Following the Grafana example above, a completed alloy.env file will look like this Set docker to run as a systemd service with sudo systemctl enable docker if you didn't already in the installation. Test the stack with sudo docker compose up Navigate to your Grafana Node Exporter Full dashboard (it may take a reselect of Datasource to populate) but you should now see the dashboard with all its data. Ctrl-C out of that and then start up the stack in the background with sudo docker compose up -d","title":"Observability Stack"},{"location":"software/server_setup/#pi-ssh","text":"The last thing will be to configure easy access to the Raspberry Pi's SSH. We can do that by creating a passwordless SSH key and installing it on the pi. We're going to be behind ssh anyway, and the Pi isn't public-facing, so this is a safe thing to do. On the server, generate a new ssh keypair with ssh-keygen -t rsa Then, install it on the pi with ssh-copy-id pi@192.168.0.2 Finally, create an SSH config that automatically supplies the hostname and user: Create a file on the GReX server in ~/.ssh/config with the contents Host pi Hostname 192.168.0.2 User pi Now you can test the easy connection with ssh pi All done! We should now be ready to run the pipeline!","title":"Pi SSH"},{"location":"software/snap/","text":"SNAP Configuration and Bringup The digital backend to the GReX system is a SNAP board from the CASPER group at Berkeley. This board contains the analog to digital converters and Xilinx FPGA to perform the digitization and F-engine components of the system. The setup and configuration of this board has seemingly never been well documented, so we'll try to make it as painless as possible here. c The FPGA Simulink model is stored here with the latest releases found here . Grab the latest fpg file, and you're good to go - no reason to recompile it. SNAP Golden Image To document the steps that are performed before shipping a box, here are the steps to program the golden (bootstrapping) image: Set switch S1 on the SANP so that switches 2 and 5 are set to on. (In the on position, the switches are moved towards the edge of the PCB). The other switches on S1 should be off. This allows the SNAP to boot from the nonvolatile flash memory. To flash this storage device, you need the free Vivado Lab Edition (VLE) and the expensive Xilinx Platform Cable. After launching VLE, click Open hardware manager . After this, a green bar at the top should ask you to autoconnect. Assuming the platform cable is plugged into the SNAP and everything is powered on, click on that. It should connect and show the xc7k160t FPGA in the hardware section on the left. In VLE, under tools go to Add configuration memory device . Use the memory device n25q256-3.3v-spi-x1_x2_x4 . It will then ask if you want to program the memory now, hit Ok . Using the golden image for the configuration file here , program the entire configuration memory device. Unplug programmer before rebooting. All done! A reboot should start blinking some LEDs and the ethernet should get a DHCP address (if it's plugged in). TAPCP and the Raspberry Pi The gateware we've built for the SNAP board includes a small CPU called the MicroBlaze that hosts a small webserver to deal with runtime interactions with FPGA memory. This server gets an IP address from a DHCP server running on the main capture computer. This interface can also be used to reprogram the SNAP if the gateware changes. By deafult, we'll ship SNAP boards that have the GReX gateware preprogramed, but it's always possible to reprogram it. This interface is over the UDP protocol TFTP, where the folks at CASPER have wrapped reading and writing files as the interaction to both the flash and FPU memory. We've written a wrapper to the so called \"TAPCP\" protocol here . It is with this library that the packet capture code informs the SNAP when to activate timing signals. FPGA Clocks, References, PPS, Synthesizers The FPGA needs an external clock source, which is being provided via one of the outputs of the Valon synthesizer in the box. Additionally, the board needs \"pulse per second\" (PPS) ticks, which come from the GPS reciever. The Valon synthesizer has two outputs but can lock both to an external reference. The GPS receiver in the box provides this (10 MHz) alongside the PPS signal. SNAP Operation Controlling the Gateware UDP Payloads Once the pipeline is running, the SNAP will be streaming high-speed data over UDP to the processing server. These UDP frames contain one timestep of both polarizations in 8+8 bit complex data. Additionally, the payload starts with a 64-bit header that is the number of frames since the first one. Assuming you record the time when you trigger the start of packets, you can then work out the associated timestamp of the packet using the fact that every packet arrives in a 8.192us cadence. The format of the payload itself has a C struct compatible layout. +-----------------------------------------------------------------------+ | Timestamp (unsigned 64-bit, little-endian integer) | +-----------------------------------------------------------------------+ | A0000R | A0000I | A0001R | A0001I | A0002R | A0002I | A0003R | A0003I | | ... | | A2044R | A2044I | A2045R | A2045I | A2046R | A2046I | A2047R | A2047I | +-----------------------------------------------------------------------+ | B0000R | B0000I | B0001R | B0001I | B0002R | B0002I | B0003R | B0003I | | ... | | B2044R | B2044I | B2045R | B2045I | B2046R | B2046I | B2047R | B2047I | +-----------------------------------------------------------------------+ Where XnnnnR + XnnnnI *j is the complex voltage data of polarization X (either A or B), channel nnnn where 0000 is 1530 MHz and 2047 is 1280 MHz. The components are signed bytes.","title":"SNAP"},{"location":"software/snap/#snap-configuration-and-bringup","text":"The digital backend to the GReX system is a SNAP board from the CASPER group at Berkeley. This board contains the analog to digital converters and Xilinx FPGA to perform the digitization and F-engine components of the system. The setup and configuration of this board has seemingly never been well documented, so we'll try to make it as painless as possible here. c The FPGA Simulink model is stored here with the latest releases found here . Grab the latest fpg file, and you're good to go - no reason to recompile it.","title":"SNAP Configuration and Bringup"},{"location":"software/snap/#snap-golden-image","text":"To document the steps that are performed before shipping a box, here are the steps to program the golden (bootstrapping) image: Set switch S1 on the SANP so that switches 2 and 5 are set to on. (In the on position, the switches are moved towards the edge of the PCB). The other switches on S1 should be off. This allows the SNAP to boot from the nonvolatile flash memory. To flash this storage device, you need the free Vivado Lab Edition (VLE) and the expensive Xilinx Platform Cable. After launching VLE, click Open hardware manager . After this, a green bar at the top should ask you to autoconnect. Assuming the platform cable is plugged into the SNAP and everything is powered on, click on that. It should connect and show the xc7k160t FPGA in the hardware section on the left. In VLE, under tools go to Add configuration memory device . Use the memory device n25q256-3.3v-spi-x1_x2_x4 . It will then ask if you want to program the memory now, hit Ok . Using the golden image for the configuration file here , program the entire configuration memory device. Unplug programmer before rebooting. All done! A reboot should start blinking some LEDs and the ethernet should get a DHCP address (if it's plugged in).","title":"SNAP Golden Image"},{"location":"software/snap/#tapcp-and-the-raspberry-pi","text":"The gateware we've built for the SNAP board includes a small CPU called the MicroBlaze that hosts a small webserver to deal with runtime interactions with FPGA memory. This server gets an IP address from a DHCP server running on the main capture computer. This interface can also be used to reprogram the SNAP if the gateware changes. By deafult, we'll ship SNAP boards that have the GReX gateware preprogramed, but it's always possible to reprogram it. This interface is over the UDP protocol TFTP, where the folks at CASPER have wrapped reading and writing files as the interaction to both the flash and FPU memory. We've written a wrapper to the so called \"TAPCP\" protocol here . It is with this library that the packet capture code informs the SNAP when to activate timing signals.","title":"TAPCP and the Raspberry Pi"},{"location":"software/snap/#fpga-clocks-references-pps-synthesizers","text":"The FPGA needs an external clock source, which is being provided via one of the outputs of the Valon synthesizer in the box. Additionally, the board needs \"pulse per second\" (PPS) ticks, which come from the GPS reciever. The Valon synthesizer has two outputs but can lock both to an external reference. The GPS receiver in the box provides this (10 MHz) alongside the PPS signal.","title":"FPGA Clocks, References, PPS, Synthesizers"},{"location":"software/snap/#snap-operation","text":"","title":"SNAP Operation"},{"location":"software/snap/#controlling-the-gateware","text":"","title":"Controlling the Gateware"},{"location":"software/snap/#udp-payloads","text":"Once the pipeline is running, the SNAP will be streaming high-speed data over UDP to the processing server. These UDP frames contain one timestep of both polarizations in 8+8 bit complex data. Additionally, the payload starts with a 64-bit header that is the number of frames since the first one. Assuming you record the time when you trigger the start of packets, you can then work out the associated timestamp of the packet using the fact that every packet arrives in a 8.192us cadence. The format of the payload itself has a C struct compatible layout. +-----------------------------------------------------------------------+ | Timestamp (unsigned 64-bit, little-endian integer) | +-----------------------------------------------------------------------+ | A0000R | A0000I | A0001R | A0001I | A0002R | A0002I | A0003R | A0003I | | ... | | A2044R | A2044I | A2045R | A2045I | A2046R | A2046I | A2047R | A2047I | +-----------------------------------------------------------------------+ | B0000R | B0000I | B0001R | B0001I | B0002R | B0002I | B0003R | B0003I | | ... | | B2044R | B2044I | B2045R | B2045I | B2046R | B2046I | B2047R | B2047I | +-----------------------------------------------------------------------+ Where XnnnnR + XnnnnI *j is the complex voltage data of polarization X (either A or B), channel nnnn where 0000 is 1530 MHz and 2047 is 1280 MHz. The components are signed bytes.","title":"UDP Payloads"},{"location":"software/stack/","text":"Software Stack There are several moving parts to this whole project, most of which are organized in our GitHub organization . The notable pieces of software are: snap_bringup - SNAP bringup and configuration T0 - UDP Packet capture and exfil to heimdal heimdall (T1) - Our fork of the pulse detection pipeline which removes clustering and RFI excision T2 FrontendModule - Hardware and software design for the Frontend Module (FEM) These are supported by some fundamental libraries sigproc_filterbank - A rust library for reading/writing SIGPROC filterbank files psrdada-rs - A rust library for interacting with PSRDADA buffers casperfpga_rs - A rust library for interacting with the SNAP board over TAPCP","title":"Software Stack"},{"location":"software/stack/#software-stack","text":"There are several moving parts to this whole project, most of which are organized in our GitHub organization . The notable pieces of software are: snap_bringup - SNAP bringup and configuration T0 - UDP Packet capture and exfil to heimdal heimdall (T1) - Our fork of the pulse detection pipeline which removes clustering and RFI excision T2 FrontendModule - Hardware and software design for the Frontend Module (FEM) These are supported by some fundamental libraries sigproc_filterbank - A rust library for reading/writing SIGPROC filterbank files psrdada-rs - A rust library for interacting with PSRDADA buffers casperfpga_rs - A rust library for interacting with the SNAP board over TAPCP","title":"Software Stack"}]} \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index c25c6c7..e5d5015 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ diff --git a/software/operation/index.html b/software/operation/index.html index 2543108..86c76de 100644 --- a/software/operation/index.html +++ b/software/operation/index.html @@ -838,7 +838,7 @@

H1 Example

Using the requests library in Python, we can perform the HTTP request to the database. 

import requests
 import time
-import numpy as npc
+import numpy as np
 
 # Get the current (UNIX) time
 now = time.time()