Commissioning records: 2025
Summary
- [Enter overarching points from the year here]
October 10, 2025
Troubleshooting: high pressures in RSoXS chamber
On ~October 9, 2025, it was noted that the pressures in the RSoXS main chamber and load lock were higher than usual. In 2024, the pressures in the main chamber and load lock were ~5.5e-9 and ~2.2e-9 Torr, respectively. However, it was noted on October 9, 2025 that these pressures were 6.1e-8 and 3.4e-8, respectively. The pressure archive across 2024 and 2025 is snown below. Throughout 2024 and prior to July 2025, the baseline pressure was below 1e-8 Torr. The steps in the baseline correspond to the transitions between the user cycle operations and shutdown, during which the downstream M4 oxygen bleed would be turned off. At the beginning of July, not only did the pressure steadily rise, but also the pressure readings became significantly noisier. The solid sample load lock had not been vented (and subsequently pumped down) while being isolated from the main chamber between June 9, 2025 and August 8, 2025. The next time it was vented and then pumped down was September 13, 2025. Based on this information, it seems unlikely that the rising pressure is caused by outgassing of an internal contaminant.
On January 23, 2025, the main chamber pressure had risen similarly due to a turbo pump controller failure, and isolating that turbo pump improved pressures temporarily. To determine if the high pressure and noise were caused by malfunctioning turbo pumps, each pump was isolated. During this troubleshooting on October 14, 2025, the RSoXS chamber was already isolated from the rest of the beamline by having GV27A and GV28 shut. The WAXS Greateyes camera was also switched off and the sensor warmed up to avoid having contaminants condense on it during periods of higher chamber pressure. Up to 11:00 am, both main chamber pumps were open and the solid sample load lock turbo pump were exposed to the main chamber. At 11:00 am, the gate valve between the solid sample load lock and main chamber was closed. The pressure in the solid sample load lock (XF:07IDB-VA:2{RSoXS:LL-CCG:1}P:Raw-I) immediately decreased, and the noise was eliminated. The pressure in the main chamber (XF:07IDB-VA:2{RSoXS:Main-CCG:1}P:Raw-I) rose slightly, and the noise was reduced for a brief moment, after which the noise returned. At 1:00 pm, the gate valve to the solid sample load lock was opened, and the main chamber top turbo pump was isolated. The pressure rose significantly and remained noisy. At 2:00 pm, the main chamber top turbo pump was opened, and the bottom turbo pump was isolated. The pressure and noise remained similar. At 3:15 pm, the bottom turbo pump was opened, and the gate valve connecting the main chamber and solid sample load lock was closed. The noise seemed to become less freuqent initially, but it resumed later. The RSoXS station was left in this state overnight, after which the load lock gate valve was opened at 8:00 am on October 15, 2025. The main chamber pressure slightly decreased, and the noise remained similar.
Based on these tests, it is unlikely that there are issues with any of the three tested turbo pumps, as isolating each of them individually did not improve the pressure or noise in the main chamber. Also, the source of the increased pressure originates from the main chamber, not the load lock. It is unclear if the temporary reduction in noise in the main chamber pressure after closing the solid sample load lock at 11:00 am and 3:15 pm on October 14, 2025 was necessarily caused by the mechanical action of closing the gate valve (e.g., mechanical perturbation of the adjacent flanges) or if it is just a coincidence. This reduction in noise was not observed after opening the load lock gate valve at 1:00 pm on October 14, 2025 or at 8:00 am on October 15, 2025.
Around 3-5 years ago, a similar pressure rise along with pressure spikes occurred as a result of leaks from the WAXS camera assembly (https://wiki-nsls2.bnl.gov/beamline7ID1/index.php?title=Commissioning_record:_2024_and_earlier#~2020-2022). To investigate this possibility, a helium leak test was performed on the RSoXS chamber on October 15-16, 2025. To set this up, a turbo pump cart was connected to a port on the downstream inboard side of the RSoXS chamber. This port has a VAT series 45 valve. The turbo cart was connected to a Pfeiffer Vacuum Adixen ASM340 leak detector. Prior to opening the VAT series 45 valve to the main chamber, the main chamber pressure was monitored during the turbo cart pumpdown, and the pressure remained fairly constant; the VAT series 45 valve are not always reliable, and if the main chamber pressure had decreased during the pumpdown, that would have indicated that the valve was a source of leaking.
To ensure that the input helium would not get diluted and only flow to the helium leak detector, the RSoXS turbo pumps and turbo pump cart were shut off. During these times, the chamber pressure rose to 3.8e-5 Torr and likely would have risen higher if kept in this state for several hours. During a similar test on September 1, 2023 (https://wiki-nsls2.bnl.gov/beamline7ID1/index.php?title=Commissioning_record:_2024_and_earlier#August_28,_2023), it was reported that isolating the RSoXS turbo pumps overnight only caused the pressure to rise to 8.3e-8 Torr. It is unclear if the RSoXS chamber was similarly isolated or if it was being pumped down by neighboring sections of the beamline. If it were the former, then the rapid rise to 3.8e-5 Torr is suggestive of a large leak.
As was found in the past, directly spraying helium onto various locations in the RSoXS camera assembly did not show any detectable leaks. However, after sealing the assembly using a garbage bag and tape and then filling this pocket with helium, a steady leak was detected. It was attempted once more to use a longer wand to spray helium to localized spots in the RSoXS camera assembly, but a specific leak location was not found. The bagged helium leak test was only performed with the WAXS camera extended into the beam path, but the direct spraying was attempted with the WAXS camera in its extended and retracted configurations.
For the sake of completeness, the helium leak test was performed on the entire chamber. A small leak is suspected on the weld seam of the outboard viewport that is downstream of the TEM load lock. The results were not consistent/reproducible, but the area was scanned multiple times, and several of the scans showed a small leak. A second small leak is suspected on the TEY signal feedthrough on the top of the solid sample manipulator. It is possible that the ceramic contact of the feedthrough has developed a small leak. In the longer term, it would be good to investigate these leaks as well.
While performing these leak tests, it was noted that the pressure baseline rose slightly and the pressure spikes became more frequent and larger in amplitude while the WAXS camera was extended into the beam path. This issue was reproduced in the plot below. With the WAXS camera still switched off and warmed up, the WAXS camera was extended into the beam path prior to 7:30 am on October 17, 2025. After this, the WAXS camera was fully retracted at 7:34 am on October 17, 2025; extended back into the beam path at 7:47 am on October 18, 2025; fully retracted at 8:47 am on October 19, 2025; and extended back into the beam path at 7:10 am on October 20, 2025. The main chamber pressure baseline remained roughly constant at ~7e-8 Torr. However, the solid sample load lock shows a noticeably lower baseline when the WAXS camera is retracted than when it is extended into hte beam path. For both pressure readouts, there are pressure spikes in both configurations. However, the spikes have slightly lower frequency and amplitude when the WAXS camera is retracted. These results may suggest that the WAXS camera bellows may be one of the sources of leaking in the chamber. Another possibility is that the source of the leak has been moved such that the likelihood of the contamination moving into the pressure gauge changes. It is unlikely that the translation of the camera directly imapcts the indium seal, which is a known source of leaking; e.g., the compression/extension of the bellows should not change the pressure applied to the indium seal because the seal is entirely supported by the camera and not the bellows/flange. These tests are continuing to be monitored and repeated to ensure reproducibility.
Following the tests above, a residual gas analyzer (RGA) is being installed onto the main chamber to scan the species of contaminants. Scans will be run with the WAXS camera extended and retracted to determine if there are any differences in the contaminant species. Also, scans will be run with the WAXS camera switched off as well as turned on with cooling/heating at different temperatures to see if the pressure and contaminant species changes. These tests will give more information on whether the leak is coming from the indium seal or elsewhere. Helium leak tests also will be re-run at these configurations to quantify a leak rate.
To set up for these tests, gate valve 2 (GV2) was closed with the help of the vacuum group and control room. Permissions were also obtained to manipulate the VAT series 45 valve on the downstream inboard port of the RSoXS main chamber that is under configuration control. After installing the RGA, a baseline scan was performed without being exposed to the main chamber. Mass 18, corresponding to water, was the largest peak. The RGA is was baked out while still being isolated from the main chamber. The main chamber pressure was monitored to ensure that the bakeout does not impact the main chamber.
Below is the baseline scan after baking out and degassing the isolated RGA. The pressure readout on the turbo cart was 2.0e-8 Torr. For reference, RGA is not performed with a turbo cart. Ideally, an ion pump is used, which can consume contaminant species to produce a cleaner baseline signal. However, an ion pump was not available to these tests, so the baseline is subject to any contamination present in the turbo cart.
Then, the RGA was exposed to the main chamber while the WAXS camera was extended into the beam path, the camera was switched off, and the solid sample load lock was connected to the main chamber. The pressures in the main chamber and solid sample load lock were 6.5e-8 Torr and 4.0e-8 Torr, respectively. The mass 18 water peak showed a significant increase, while the other peaks remained fairly steady. So far, it seems unlikely that there is an air leak, as the peaks for nitrogen (mass 28), oxygen (mass 32), and argon (mass 40) have not shown a significant increase.
On October 28 at 8:34 am, the WAXS camera was fully retracted, and new RGA scans were started. The WAXS camera was extended again into the beam path on October 28 at 8:28 pm. The WAXS camera was retracted out of the beam path on October 29 at 8:25 am. Finally, the WAXS camera was extended into the bam path on October 29 at 8:32 pm. The RGA scans for these two configurations appear almost identical, as shown below. The pressure archives during these tests are also included below.
On October 30-31, another thorough helium leak test was performed by the Vacuum Group. The starting pressures at 9:00 am on October 30 was 7.4e-8, 4.5e-8, and 1.5e-8 Torr in the main chamber, load lock, and turbo cart. First, the chamber was leak tested with the WAXS camera assembly isolated in a plastic bag with nitrogen flow to keep that section clean. There were notable pressure spikes after the nitrogen flow started, though this is likely due to the random nature of the spikes rather than the presence of nitrogen. Nevertheless, the nitrogen flow was reduced. The leak test was performed once with the RSoXS turbo pumps connected to the main chamber and then once with those turbo pumps isolated. In both cases, the load locks and main chamber other than the WAXS camera assembly showed no significant leaks. The solid sample load lock had a leak rate of 3.8e-12 Torr L/s at a chamber pressure of 4.7e-8 Torr. After the load lock was confirmed to have no significant leak, the gate valve between the load lock and main chamber was closed. The outboard viewport on the main chamber next to the TEM load lock had a loose bolt, but no detectable leak was found there.
Initially, a bagged leak test was performed on the WAXS camera assembly with the camera fully extended into the beam path at ambient temperature and with the RSoXS turbo pumps connected to the main chamber. After flowing in helium into the bag for 30 s, a rise in the helium leak was detected, and the maximum leak rate was 2e-9 Torr L/s. Then, the RSoXS turbo pumps were isolated, and this test was performed again. The chamber pressure rose to 3.5e-5 Torr during this time. Then, the camera was retracted at 2:45 pm on October 31, 2025, and a similar leak rate was calculated; the bag setup had to be redone, as the camera assembly did not have the clearance to be retracted with the bag sealed on. The helium pressure over time is shown below for both cases. For the retracted case, the base leak rate was 8.0e-10 Torr L/s. In both cases, the leak rate was on the e-9 Torr L/s scale, and there were no significant differences between the two camera configurations.
| Time (min:sec) | He leak pressure (Torr) | Time (min:sec) | He leak pressure (Torr) | |
|---|---|---|---|---|
| 00:00 | 2.0e-9 | 00:00 | 6.5e-10 | |
| 01:30 | 2.2e-9 | 01:30 | 6.8e-10 | |
| 03:00 | 2.6e-9 | 02:00 | 8.0e-10 | |
| 04:00 | 3.4e-9 | 03:00 | 1.0e-9 | |
| 05:00 | 5.0e-9 | 05:00 | 1.5e-9 | |
| 07:00 | 6.7e-9 | 06:00 | 2.5e-9 | |
| 08:00 | 6.6e-9 | 08:00 | 4.6e-9 | |
| 09:00 | 7.0e-9 | 10:00 | 6.1e-9 | |
| 10:00 | 7.6e-9 | 11:00 | 6.8e-9 | |
| 11:00 | 8.0e-9 | 12:00 | 7.2e-9 |
Next, the effects of sensor cooling were investigated. The camera cooling had been switched off during all tests thus far. During this time, the sensor temperature was 17 °C, and the hot side temperature (near the indium seal) was 14 °C. On November 1, 2025 at 8:35 am, the cooling was switched on with a setpoint of -80 °C. Within ~15 min, the sensor temperature reached -79 °C, and the hot side was 17 °C. During the next several hours, the chamber pressure baseline decreased slightly, and the frequency of pressure spikes also was reduced. On November 3, 2025 at 7:50 am, the cooling was disabled. During the next several hours, the chamber pressure baseline returned to its original level before sensor cooling was enabled. Several negative pressures spikes occurred, but positive spikes were significantly reduced. On November 4, 2025 at 7:06 am, cooling was enabled again with a setpoint of -100 °C. However, the sensor and hot side temperatures still equilibrated at -79 °C and 17 °C, respectively. During the next several hours, there were chamber pressure oscillations at significantly reduced frequency. The chamber pressure would decrease for some time until a spike brought it back up to its pre-cooling level. On November 5, 2025 at 4:53 pm, the cooling was disabled. On November 6, 2025 at 10:12 am, cooling was enabled with a setpoint of -100 °C.
October 1, 2025
Beamline shutdown
Due to the government shutdown, NIST beamlines were shut down to a safe configuration. During this process, cooling for various pieces of hardware was turn off, though the RSoXS Greateyes WAXS camera was kept powered on with cooling to run tests with dark images. High-voltage equipment was shut off in downstream NEXAFS stations. Oxygen bleed was turned off for M1, M2/PGM, M3, and M4. For the oxygen tank that supplies the bleed for M2/PGM and M3, upstream pressure was noted as 15.6 psia; this and the other upstream pressures should be checked when the oxygen bleeds are turned back on to ensure that there are no leaks in the upstream lines. Finally, the gate valves were closed such that the various segments of the beamline remain isolated in case a vacuum failure were to occur while NIST staff are unable to monitor the beamline. The pressures throughout the beamline at the beginning of the shutdown are shown below for reference. Prior to opening the gate valves it should be verified that none of the pressures have increased significantly.
September 29, 2025
Troubleshooting: energy calibration and signal quality
The LARIAT1 station was used during this beam time using the 1200 l/mm PGM. Reference scans were run at the carbon, oxygen, and fluorine K-edges. The I0 signals from these scans show the expected increase in flux with energy and good signal quality. This indicates that the issues encountered with reference open-beam scans using the RSoXS 250 l/mm PGM during the August 4, 2025 and September 10, 2025 beam time are not universally seen at all PGMs and CFF values.
During future RSoXS beam times, it would be good to run a comparison between the RSoXS 250 l/mm PGM and the 1200 l/mm PGM to understand whether there is something physically damaged on the RSoXS 250 l/mm PGM or if it is the energy calibration method used at RSoXS that is producing bad calibrations. It would also be good to run open-beam energy scans using the 1200 l/mm PGM to check if the contamination seen on the RSoXS 250 l/mm PGM (or M2) is also present on the 1200 l/mm PGM.
September 23, 2025
Prefect upgrades
Following up from August 29, 2025, the Prefect3 workflow systems have not been reliably executing workflows, so all beamlines were rolled back to the Prefect2 system. Scan IDs 102514-102515 were run as tests after the rollback, and the workflows succeeded. The Prefect workpool also has rolled back to the rsoxs-work-pool in the dev workspace.
September 13, 2025
Sample position calibration
The corresponding fiducial scan IDs for the bar used during this beam time are 101237-101246. The bottom fiducials -90 and 90 degree positions needed manual calibration.
RE(find_fiducials(f2=[-5.025, 0.47509999999999764, 3.7, -2.2216650000000016], y2=3.5667649999999753, f1=[-2.016085000000004, -0.07313500000000062, 0.809829999999998, -1.7140900000000059],y1=-186.87915500000003))
correct_bar([3.5522599999999898, -5.137, 1.171945000000001, -1.138, -2.5627950000000013, -187.02401500000002, -1.9843150000000023, 0.11966499999999769, 0.8168949999999953, -1.3376599999999996], include_back=False)
Troubleshooting: Solid sample up-down motor
September 10, 2025
Vacuum failure in downstream portion of SST1 beamline
On September 10, 2025 afternoon, there was a sudden vacuum failure in which multiple pumps downstream of RSoXS (close to the LARIAT1 station) threw errors adn stopped running. As a result, the pressure rose in those sections, and several shutters and gate valves shut automatically to isolate those regions. The cause is being investigated. The central controller for these pumps is dead, which is likely the cause of the pump errors. It may have been caused by a fuse tripping off.
External pump carts were connected to the beamline to pump down the affected sections. However, even after the downstream sections were pumped to sufficient pressure, the necessary gate valves and shutters for RSoXS operation could not be opened via software in phoebus. This is being investivated further in this slack thread: https://nsls2.slack.com/archives/C01Q3BYKFRD/p1757593998681819
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.3 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.722 | 0.35 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -5 | -5 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 5 | 5 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -5 | -5 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | 0.5 | 1.3 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.57 | 0.57 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.2 | 24.2 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.84 | 7.83 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
A thorough beam alignment was not performed. Instead, the values used during the prior beam time were used as a starting point, and then the M3 pitch and front-end slit v center were adjusted to increase flux on I0.
Initially, there were communication errors encountered on the I400s, but these were due to a disconnected power cable on the downstream Moxa unit.
The WAXS beamstop position was adjusted by taking measurements on the WAXS CCD camera (scan IDs 100661-100670) Further checks for the beamstop position were performed in scan IDs 100747-100749.
Beam centering was performed based on scan ID 100879, which is a scattering image of SBA-15 at 291.65 eV, 90° polarization. The beamstop was masked with a quadrilateral polygon with the following vertices in the x (inboard-outboard) and y (up/down) coordinates: ["{\"x\":{\"0\":535.415,\"1\":511.344,\"2\":-0.476,\"3\":-0.476,\"4\":535.415},\"y\":{\"0\":477.418,\"1\":446.074,\"2\":872.214,\"3\":948.916,\"4\":477.418}}"]. The beam center was found to be [474, 502] (pixels in the inboard-outboard and up-down directions, respectively), and sample-to-detector distance was 31.960803248151926 mm.
Sample position calibration
The same sample bar from November 8, 2024 was used. This bar has been in the chamber since the August 4, 2025 beam time, and all position calibrations are the same.
Energy calibration and signal quality adjustments
Multiple energy calibrations were performed during this beam time because the flux/signal was likely compromised by several factors:
- Contamination on the PGM. This was first reported during the August 4, 2025 beam time.
- Communication glitches with Tiled. These glitches resulted in explicit ReadTimeout errors, and they also resulted in certain datapoints being skipped/compromised when there were no explicit errors. Details are included in the following slack threads:
- Detector signal processing issues. The WAXS beamstop photodiode and generally other detectors processed by the I400 electrometers often show unreliable signals and cross-talk. The SRS570 current pre-amplifiers initually were on sub-optimal settings.
The table below summarizes the energy calibration attempts and associated scan IDs. All scans were run with m_order = 1 for all CFFs. After grating and mirror offsets were tuned, a standard carbon-K-edge energy scan was performed on HOPG at 90° polarization and with the sample bar rotated to 70° to get maximum beam footprint. These scans were run in ascending and descending energy to check for reproducibility.
| PGM optimization scan IDs | CFFs | Mirror 2 angles | Grating angle with maximum signal | Offset angles [M2, grating] | HOPG energy scan IDs | Open-beam energy scan IDs |
|---|---|---|---|---|---|---|
| 100578-100592 | [1.45,1.5,1.55,1.6,1.65] | [-3.0810894331520444, -2.9530220058665435, -2.8433762345820455, -2.748519194291042, -2.6663929189960456] | [-3.6281320559809984, -3.525168209648001, -3.438296605486997, -3.364780385509995, -3.3016999243995] | [-5.492e-03, 1.467e-02] | 100593-100594 | |
| 100642-100656 | [1.45,1.6,1.65,1.5,1.55] | [-3.081100383149554, -2.7489255185115553, -2.6662363567220524, -2.952721861645557, -2.8433791379555586] | [-3.647422633273223, -3.3840663527572232, -3.321031866368223, -3.543672480901222, -3.457230616753222] | [-9.184e-03 -7.319e-03]
Calculated manually |
100657-100658 | 100671-100672 |
| 100675-100689 | [1.45,1.5,1.55,1.6,1.65] | [-3.0810930626883746, -2.9527261920253736, -2.8433768466343707, -2.7484575288263713, -2.6663940339623764] | [-3.647525914926142, -3.543748689017143, -3.457271076065645, -3.3833438496221433, -3.32062988708914] | [ 2.617e-03 2.090e-03] | 100690-100691 | 100692-100693 |
| 100709-100738 | [1.35,1.4,1.45,1.5,1.55,1.6,1.65,1.7,1.75,1.8] | [-3.4213818791967228, -3.2337517065927273, -3.0810956403812284, -2.952724578768226, -2.843377580309223, -2.748492628291224, -2.666393845628228, -2.593557397741229, -2.5280865444792227, -2.4706214953642274] | [-3.930190036826623, -3.7728195709601238, -3.6472818849136246, -3.5437369633631235, -3.4572531478056234, -3.38320128250912, -3.3202360306141188, -3.2648510757126203, -3.2174974897531214, -3.1746720995401247] | [ 3.875e-03 3.786e-03] | 100739-100740 | 100741-100742 |
| 100882-100911 | [1.35,1.4,1.45,1.5,1.55,1.6,1.65,1.7,1.75,1.8] | [-3.421381281692831, -3.234389013588334, -3.0810991500083347, -2.9527247356353357, -2.8428250346903354, -2.74896011799283, -2.666392074658333, -2.5929918182678335, -2.5284106617813293, -2.4706154496253347] | [-3.931026293357604, -3.774145372360607, -3.648013325785108, -3.5444337031686075, -3.4575721572876077, -3.3845825720876093, -3.3208139150776077, -3.265974621866107, -3.217313124230607, -3.1758386867501045] | [ 4.432e-03 3.451e-03] | 100912-100913 | 100914-100915 |
| 100934-100951 | [1.35,1.4,1.45,1.5,1.55,1.6] | [-3.4213854912073316, -3.2343885711483367, -3.081096067269833, -2.952720311792831, -2.8433739419763313, -2.748931093273832] | [-3.9305204663904476, -3.7737094923489494, -3.647804679082448, -3.5442997391949476, -3.4578644548384503, -3.3845799943964465] | [-2.283e-02 -1.861e-02] | 100952-100953 | 100954-100955 |
PGM optimization scan IDs 100578-100592 and the associated HOPG scan IDs 100593-100594 were performed with both I0 and the sample current processed using an I400 electrometer. The HOPG scans showed good energy resolution, but forward and reverse energy directions did not overlap in intensity in the resonant regions.
Later, the I0 mesh and TEY signals were switched to SRS570 current pre-amplifiers that fed into ADC channels. The I0 mesh was connected to SRS570 #13 (numbers are purely to distinguish different SRS570 units) behind the ucal station using a single long BNC cable, and this SRS570 unit was connected to ADC3 channel 2 (XF:07ID-BI[ADC:3-Ch:2]Volt). The SRS570 settings used were: no bias voltage, filter type = 12 dB high pass, filter frequency = 100 Hz, gain mode = low noise, input offset = off, sensitivity = 1x1 nA/V, power = line. The TEY signal was connected to SRS570 #12/16 on a rack near the ADC units located roughly behind the PEY NEXAFS chamber using a single long BNC cable, and this SRS570 unit was connected to ADC3 channel 1 (XF:07ID-BI[ADC:3-Ch:1]Volt). The SRS570 settings used were: no bias voltage, filter type = none, filter frequency = none, gain mode = low noise, input offset = off, sensitivity = 1x10 nA/V, power = line. After this, it was attempted to calibrate PGM offsets again, but the full sweep of CFFs was interrupted multiple times by ReadTimeout errors from Tiled. The grating positions found from scan IDs 100642-100656 were gathered manually to calculate offset angles. The HOPG and open-beam energy scans (scan IDs 100657-100658 and 100671-100672, respectively) taken after this PGM calibration attempt were very noisy. For the WAXS photodiode signal, specifically, it was initially assumed that the noisiness was due to faulty cable connections and/or I400 artifacts, as has occurred several times in the past. However, the signal was only noisy in the presence of beam but not when the chamber light was turned on; this, combined with the noisy HOPG signal, suggested that the PGM calibration may have issues that are causing the beam to be unstable. It was suspected that even the scans that did not explicitly terminate due to ReadTimeout errors were likely compromised from the Tiled issues. Another likely issue is the PGM contamination
The PGM was calibrated once (scan IDs 100675-100689), and no explicit ReadTimeout errors occured during the scans. The associated HOPG and open-beam scans were slightly less noisy, but still, the signal quality was not optimal.
The PGM was calibrated again (scan IDs 100709-100738), this time using a larger range and number of CFFs. The intention was that more datapoints for the grating maxima would result in a better fit of the mirror and grating offsets. The associated HOPG scans did not show significant improvement, but the open-beam scans were slightly less noisy. With the same energy calibration, the SRS570 settings were adjusted, which resulted in smoother HOPG profiles (scan IDs 100743-100744). The new SRS570 settings used for the I0 mesh were: no bias voltage, filter type = 6 dB high pass, filter frequency = 100 Hz, gain mode = low noise, input offset = positive 1x1 pA, sensitivity = 1x1 nA/V, power = line. The new SRS570 settings used for the TEY signal were: no bias voltage, filter type = 6 dB high pass, filter frequency = 100 Hz, gain mode = low noise, input offset = positive 1x1 pA, sensitivity = 1x10 nA/V, power = line.
At this point, the energy calibration was assessed via Scan IDs 100878-100881, in which SBA-15 scattering images were captured over a large energy range at varying polarizations. The actual energies were extracted from the Bragg peak radius based on the sample-to-detector distance that was calculated above at 291.65 eV; see March 8, 2025 for more details on the methods used. The results show a systematically increasing deviation from the setpoint energy as the setpoint energy gets further from 291.65 eV; the magnitude of the deviation seems similar to that found on March 8, 2025, suggesting that the recent PGM disruptions did not impact this aspect of the energy calibration.
Finally, another energy calibration was attempted (scan IDs 100882-100911) with the expanded range of CFFs. However, the resulting HOPG and open-beam scans looked poorer. A final energy calibration was performed (scan IDs 100934-100951), in which a limited number of CFFs were selected based on where energy resolution was better. The associated HOPG and open-beam scans looked better, but there are still major issues with the energy calibration. Open-beam scans at higher energies had similar issues of high noise and low flux, as was reported during the August 4, 2025 beam time. Additionally, energy resolution is very poor. A set of HOPG energy scans was performed (scan IDs 100956-101009) for a range of slits1 vsizes between 0.01 mm and 10 mm. At higher slit aperture sizes, the signal got saturated; this can be resolved in the future by reducing the gain on the SRS570 that is connected to the TEY signal. At low slit sizes, the scans failed to reproduce the energy resolution shown on May 25, 2024 and June 5, 2024. Although the scans taken during 2024 had more datapoints, it was confirmed that reducing the point density to one-third of the original point density, which was sparser than the scans run during the current beam time, did not visually impact the energy resolution differences seen at lower slit aperture sizes. These results suggest that the poor energy resolution is another symptom of the faulty energy calibration. In 2024, these scans were run primarily to select an appropriate slits1.vsize. However, the results below suggest that these scans should be run more routinely to check energy resolution; moreover, the single HOPG scan run after energy calibration should be compared to those run during 2024 at a similar slits1.vsize to check the energy resolution.
The open-beam energy scans that were taken prior to the PGM repair (May 27, 2025; April 3, 2025; January 30, 2025) show more expected behavior, suggesting that the energy calibration issues causing the noisy, low-flux signals during recent open-beam scans likely stem from something that occurred during the PGM repair.
Based on the information above, it is hypothesized that carbon contamination on the upstream optics (e.g., August 4, 2025, File:20250809 PGMContamination.png) might distort the signal of the HOPG profile so that the grating angle at which a maximum in the raw sample current is found is not exactly at 291.65 eV. This distortion likely causes an error in the M2 and PGM angle offsets calculated and then a subsequent mismatch with the EPU gap; the flux is sensitive to the synchronization between the EPU gap and M2/PGM angles (e.g., September 17, 2024, File:20240919 EPUGapScans.png). The energy calibration is performed at 90° beam polarization. Based on the data from August 4, 2025, the contamination signal should be minimal at 291.65 eV at 90° polarization, but a small dip is still present that may be distoring the signal slightly. Based on this hypothesis, a proposed solution is to double normalize the sample current using an open-beam scan run at the same conditions. The maximum of the double-normalized signal should be less-susceptible to the distortions than the raw sample current. This method will be less automated than the current protocol, because data security will not easily allow accessing multiple scans (the HOPG scan and associated open-beam scan) within the same Bluesky run to perform double-normalization before finding the maximum. However, if this solution works after manual testing, there may be ways to incorporate it more seamlessly into the nbs-gui adn nbs-viewer workflows and use it universally across all SST beamlines.
Reference energy scans
Characterizing I0 contamination
The gold mesh TEM grids that wre characterized during the August 4. 2025 beam time were also scanned during this beam time. These grids had not been removed from the RSoXS chamber since the prior beam time, so they did not receive any air exposure. This time, the samplecurrent signal was processed using SRS570 current pre-amplifiers instead of the I400 electrometers. The as-received grid was scanned repeatedly at the carbon K-edge at 0° beam polarization at scan IDs [101042, 101043, 101044, 101045, 101080, 101081, 101082, 101083, 101118, 101119, 101120, 101121, 101156, 101157, 101158, 101159, 101223, 101224, 101225, 101226]. The grid that was UVO'ed for 90 minutes was scanned at scan IDs [101042, 101043, 101044, 101045, 101080, 101081, 101082, 101083, 101118, 101119, 101120, 101121, 101156, 101157, 101158, 101159, 101223, 101224, 101225, 101226]. The TEY signals were still very noisy even when using the SRS570 current pre-amplifier, suggesting that the primary issue is likely poor electrical contact with the sample bar. However, the TEY signal shows signifantly more resonant signal relative to that seen during the August 4, 2025 beam time. Sigificant growth was not seen across scans during the same beam time, suggesting that contamination may have accumulated between the two beam times.
Troubleshooting: WAXS Greateyes CCD camera artifacts
These measurements are a continuation of those that were performed during the August 4, 2025 beam time. SBA-15 was used as a reference sample because its scattering pattern exhibited weak anisotropy, likely due to geometric polarization; this weak anisotropy is ideal for characterizing the signal differences across the different quadrants in a 4-ADC readout setup and also signal artifacts within each quadrant.
Each scan was performed at energies [291.65, 300, 400, 500, 600] eV, and scans were performed at [0, 90, 45, 135]° polarization at each condition. Before running scans at each condition, dark_frame_preprocessor_waxs.clear() was run to clear darks such that a new dark image would be captured that is representative of the scan conditions. Scans were run in single-ADC mode for ADC E (scan IDs 100751-100754), ADC F (scan IDs 100765-100768), ADC G (scan IDs 100769-100772), and ADC H (scan IDs 100773-100776), all with 4x4 binning. Scans also were run in 2-ADC mode for ADCs E,F (2 Amps, 1 Reg, scan IDs 100859-100866) in 4x4 binning mode; more than 4 scans were required because ReadTimeout errors were encountered during this series, so some energy series were divided over multiple scans. Another set of scans were performed using 4 ADCs, but with 2x2 binning (scan IDs 100867-100876). It was attempted to perform similar scans with 1x1 binning, but the camera server would always shut off while a scan was attempted, similar to the issues occurring in 2-ADC mode during March 24, 2025; it is unclear why this issue ocurred, as 1x1 binning had been used successfully during ~March, ~June, and ~December 2024. Finally, scans were performed using the default 4-ADC and 4x4 binning mode (scan IDs 100878-100881).
August 29, 2025
Prefect upgrades
During the shutdown, upgrades were made to the Prefect infrastructure, which can apply data reduction/analysis workflows to the raw data and write out the results into a proposal directory. Now, the beamline runs containerized workflow on prefect-worker1 (previously was running on prefect-worker1, and these have been turned off). Details can be seen here: https://github.com/NSLS-II-SST/workflows-rsoxs/pull/15
After resolving some permission issues to write out to the commissioning directory, the upgrades were deployed at RSoXS. Scan IDs 98816-98819 were run as tests, and the workflows succeeded. During this migration, the Prefect workpool has moved from rsoxs-work-pool in the dev workspace to the rsoxs-work-pool-docker in the prod workspace.
August 4, 2025
PGM repairs
The cooling lines were repaired July 16-18. After this point, the PGM chamber was sealed, pumped down, and baked out during the following week. Mechanical surveys were conducted before and after baking. After the travelers were approved, beam could be sent through the PGM chamber.
With beam available, the PGM optics were tested without oxygen bleed in the PGM chamber. To avoid carbon contamination on the gold mirror stripe, M2 was initially moved to the carbon stripe, and the grating was at 1200 l/mm #2 (likely the spot that is already contaminated). After PSH4 was opened, a the pressure in the PGM chamber increased. This pressure increase could be due to the carbon stripe being hardly used and under-characterized or due to the PGM chamber being open to air for significant time during the repairs. M2 was moved to the nickel stripe, and there was also a pressure increase, though it was noted if it was a similar magnitude. The outgassing appeared to be roughly proportional to the beam flux, whether that was tuned by energy or by the beam size. The PGM pressure is being archived (PV name: XF:07IDB-VA:2{Mono:PGM-IP:1}P-I), and is shown below. After determining that the front-end slits could be opened all the way without the outgassing reaching upper pressure limits, radiation surveys radiation surveys were conducted on August 4 afternoon to close out safety permits. The PGM pressure remained in the e-8's up to and throughout August 5. The oxygen bleed was turned on August 5 ~evening, and the mirrors were moved to the gold stripe.
The PGM will need calibration, whether it is through the the standard tuning with HOPG or more involved realignment. While attempting to run an initial carbon-edge energy scan with HOPG, no features were seen. At some point on August 5, the 1200 l/mm grating was successfully calibrated to the correct energy. Two out of three spots on the grating/M2 were successfully calibrated but one spot (#2) could not be calibrated, possibly because it got contaminated during initial beam exposure without oxygen. The 250 l/mm grating (different from the RSoXS 250 l/mm grating) also was tuned successfully on August 6-7.
The RSoXS 250 l/mm grating also was tuned successfully using the automated Bluesky function in scan IDs 96353-96367. The results are the following: mirror positions [-3.081087774493689, -2.952879908473683, -2.843382119633688, -2.7483870713591827, -2.666393942545689], grating positions [-3.669807700970999, -3.5657604940629994, -3.4786824437049972, -3.4041091404094956, -3.3409962832554996], mirror angle offset 3.485e-02, grating angle offset 6.127e-03. A reference HOPG scan was taken with scan ID = 96368 at 90° polarization with sample bar rotated 70° to get maximum beam footprint and showed good resolution.
The presence of contamination on the PGM and M2 was further confirmed by open-beam scans using the RSoXS I0 gold mesh and WAXS photodiode. The photodiode signal shows a broad dip as well as sharp peaks at the carbon K-edge, suggesting that significant adventitious carbon has deposited onto the upstream optics. Notably, the contamination seems to be oriented, as the resonant peaks shift significantly with beam polarization. The normalized I0 / Photodiode signal has similar shape across all polarizations, suggesting that all polarization-dependent resonant peaks seen in the raw I0 and photodiode signals are originating from upstream contamination.
Open-beam scans also were performed at the nitrogen, oxygen, and fluorine K-edges to assess whether there may be contamination present that contains these elements. No resonant peaks or significant polarization dependence were seen. However, the beam flux is inconsistent with the expected increase in beam flux between ~200 eV and ~800 eV (also confirmed in open-beam scans taken on May 14, 2023, June 5, 2024, and July 14, 2024); e.g., the flux at the fluorine K-edge is lower than that at all other edges. It is unclear if the low flux is the cause of the noisy signals. It is also unclear if these low fluxes might be the result of contamination present on the optics that contains these elements or because of poor energy calibration (the quality of energy calibration was only assessed at the carbon K-edge). It is more likely the latter, as the former likely would have shown resonant peaks at these edges. The presence of carbon contamination shown above likely compromised the quality of the energy tuning if the resonant peaks of the contamination overlapped with the HOPG peaks. The consistent drop in at both the I0 gold mesh and WAXS photodiode suggests that it is related to upstream beam characteristics as opposed to glitches in signal quality at any of these detectors.
Open-beam scans were performed at various times throughout the beam time to assess how the raw signals and double-normalized signals evolve with time. At the carbon K-edge, the raw I0 and photodiode signal shapes appear to vary over time, suggesting that the nature of the contamination on the upstream optics may have changed, but there is no obvious decrease in the resonant peak signals. It might be necessary to expose the optics to oxygen and beam for much longer time to clean off the contamination. Also, it might be good to individually move the grating and M2 to different spots to deconvolute where the contamination is coming from.
All double-normalized signals except those from the first scan agreed. It is unclear why the first scan does not agree; according to the motor positions saved in the baseline stream, all scans were run at the same instrument configuration. For pairs of consecutive scan IDs (e.g., 96844 and 96845), the first scan was run in ascending energy order, and the second scan was run in descending energy order; the signals seem to agree in these pairs of scans, suggesting good reproducibility.
Similar open-beam scans were performed at other edges. Unlike the carbon K-edge, not all cases showed raw signal reproducibility across forward and reverse energy scans or normalized signal consistency over time. It is unclear if these results are related to the flux issues reported above.
Troubleshooting: EPU faults for polarizations above 90°
Similar to prior cases (e.g., April 3, 2025), the EPU experienced motor faults when polarizations above 90° were set and then it was attempted to set the polarization to 0° for the next scan. As before, this issue is not consistent. Several scans were run with the polarization being set to values above 90° and then being returned to values below 90° without error (e.g., scan ID 96389 onward). The fault occurred after a set of energy scans (440 eV and 560 eV) being run at 0°, 90°, 45°, and 135° in that order (scan IDs 96952-96959). Originally, Ax:BO had faulted, and the queue of scans paused and hung indefinitely after that. After stopping the queue of scans and attempting to manually move the polarization to 0°, the Ax:TI faulted (showing red on phoebus), and Ax:BO showed a warning (yellow). The controls staff attempted to reset the EPU motors, but subsequent attempts to change the polarization even at smaller increments (e.g., 135° to 130°) resulted in errors. After some attempts to reset the EPU motors, the readback on phoebus did not show errors, but neither the EPU gap nor phase could be moved after this. The insertion device team was contacted to troubleshoot further.
During the shutdown, it was attempted to reproduce the error. First, the polarization was manually moved to 0°, 45°, 90°, 135°, and then back down to 0°. Further attempts were made to move to 150° and 170° and then move to 0°. In all of these cases, the motor error was not reproduced. A loop was set to run energy scans (440 eV and 560 eV) being run at 0°, 90°, 45°, and 135° in that order to recreate the conditions that had produced the error after scan ID 96959. Before the first scan of this loop started, there was an Ax:TO error. However, after resetting the EPU motors and restarting the scans, the scans seemed to run without explicit errors for several hours (e.g., scan ID 96968-98502, 98503-98594, 98615-98641, and 98643-98815).
Although an explicit motor fault was not recreated, there seems to be another issue arising with the EPU, as seen during the final scans of the following groups of scan IDs: 98503-98594, 98603-98614, and 98820-98975. During the last scan of each series, which was running at a setpoint polarization of 135°, the scan would pause and hang indefinitely. After aborting the run engine and checking the polarization, the readback value would be at ~177° (except scan ID 98975 where different troubleshooting was performed), which was never entered as a setpoint. Subsequent attempts to manually change the polarization to 0°, 135°, etc. would result in successful motions without any motor faults. Restarting the loop of test scans would also result several scans running successfully until eventually, this issue occurs again. During its first occurrence, it was unclear whether the indefinite hanging was due to an EPU error or a network glitch, as several updates were occurring to the software infrastructure during the synchrotron shutdown. However, after a couple occurrences that would always occur at scans with a 135° polarization setpoint, and would end with the polarization readback at ~177° polarization, it seems more likely that the issue is related to the EPU. While scan ID 98975 was in a hung state, it was noted in phoebus that the actual phase offset (readback) was stuck at a different value than the setpoint phase offset. Simply clicking on the setpoint and hitting enter seemed to allow scans to resume. It is still unknown why the motors become stuck in the first place.
Another subtle issue was observed in the polarization readback values for several scans with 135° polarization setpoint. All scans with 0°, 45°, and 90° polarization setpoints showed the correct polarization readback values at both the beginning and end of the scan. However, the scans with 135° polarization setpoint often (but not always) showed a low value as the starting polarization readback and then 135° as the ending polarization readback (e.g., [ 2.74828147 134.99803089]). The occurrence of the low starting polarization readback seems random, but all occurrences of above issue where scans would hang indefinitely had a similarly low starting polarization readback value.
A spreadsheet of representative scans can be seen here: File:EPU test scan log 2.xlsx.
While manually changing polarizations, the EPU phase offset and mode values were observed. When changing polarizations between 0° and 90°, the phase mode stays in Antiparallel TI-BO. A phase of 0 um corresponds to 0° polarization at all energies. At 440 eV, for example, 90° polarization corresponds to a 29500 um phase, and 45° polarization corresponds to a 16649 um phase; the EPU gap also changes to maintain the same 440 eV energy. When the polarization crosses 90°, for example a 45° to 135° transition, the phase first goes from 16649 um to 0 um (corresponding to 0° polarization), then the mode switches from Antiparallel TI-BO to Antiparallel TO-BI, and finally the phase increases back to 16649 um in this new mode. Going from 135° to 0° polarization requires the phase to first reduce to 0°, and then the mode to switch from Antiparallel TO-BI to Antiparallel TI-BO. When the polarization is switched from 180° to 0°, only the mode switches from Antiparallel TO-BI to Antiparallel TI-BO, but the phase value remains at 0°.
It was attempted to prevent/reduce the glitch that seemed to occur during the 135° polarization scans by modifying the base_set_polarization function in the sst_base codebase such that the polarization was explicitly moved to 0°, then explicitly moved to 180° to switch the mode, and finally moved to 135°; see commits ae102b4 (https://github.com/NSLS-II-SST/sst_base/commit/ae102b40e60aeb4b0c5f1c3b03a2361c19382f09) and 3d9c1ec (https://github.com/NSLS-II-SST/sst_base/commit/3d9c1ec6af07b5cc9d370e945adb871714134e9f). A limited number of test scans were run. Although none of the 135° setpoint polarization scans had the issue of a low polarization readback starting value, a different issue arose. The scans hung much more frequently, and all were cases when the polarization was transitioning from 45° to 135°. The EPU gap and phase setpoints were 22362.31 um and 16649.34 um, respectively, but the readback values were stuck at ~27355.20 um and ~16668.60 um, respectively. When the setpoint gap value was manually entered into phoebus, the scans would resume, but they would hang soon during a future transition from 45° to 135°.
A spreadsheet of the test scans can be seen here: File:20250903 EPU test scan log.xlsx.
Beam alignment
The front-end slits, FOE slits, and M1 parameters are the same as those that were set on July 11, 2025. M3 parameters may be adjusted if needed.
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.52 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.6 | 0.5 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -5 | -5 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 5 | 5 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -5 | -5 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | 1.3 | 1.3 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.57 | 0.57 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.2 | 24.2 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.781 | 7.84 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
It is noted that EPU60 front-end slit v center was changed 0.5 because otherwise, the beam was too low on the RSoXS yag. It was attempted to raise the beam using mir1.roll based on tests during July 11, 2025, but it made the beam too thin.
Images of the beam were captured on the FS7 pink beam stop with scan ID 96283 (zero-order) and 96284 (energy nominally at 270 eV, but not calibrated yet at this grating).
Images of the beam also were captured on the RSoXS I0 YAG with scan ID 96286 (slits 1 fully open) and 96287 (slits 1 closed down to the aperture sizes used during normal operation).
The beamstop position was aligned during scan IDs 96288-96300 with Precision Slit C being opened gradually to increase beam flux. After this, images were taken with exposure times of 0.1 and 1 s. The WAXS beamstop signal was extremely finicky; changing from the 6-foot cable to the 3-foot cable that connects the beamstop to the I400 seemed to make the signal more stable, although in the past, the 3-foot cable also had similar issues.
Characterizing I0 contamination
Open-beam scans were performed with the I0 gold mesh translated to different positions to survey the contamination across the mesh. As shown in the plots below, not all regions of I0 allowed the beam to be transmitted fully to the WAXS photodiode. It might be the case that some of the highest and lowest points surveyed may be more of an opaque film rather than a transmissive mesh. The shape of the I0 vs. energy plot also appears different especially at some of the lower points (purple curves) on teh mesh, suggesting that the nature contamination is different at those locations.
The above set of scans were replotted after removing the scan IDs in which the full beam did not pass through to the WAXS photodiode. Additionally, the scans at which the I0 mesh was translated to a position of -31 mm are represented by black-colored curves; this is the position at which the RSoXS station normally operates. In all cases, the normalized signal varies across the different I0 mesh positions; this result is expected, as the different positions are expected to have differnet levels of contamination. Also, the I0 signal at a mesh position of -31 mm shows the largest magnitude of signal; this result is also expected, as contamination is greatest on this spot which receives significantly more beam exposure than any of the other spots. Notably, the dynamic range of the I0 signal at the -31 mm position is less than the dynamic range at some other signals, suggesting that the steady-state contamination may level out the dynamic range. One of the orignal concerns with the contamination on the gold mesh was that a large dynamic range caused by the contamination may compromise assumptions made about the linearity of the I0 signal.
Additionally, brand-new gold mesh TEM grids with the same mesh characteristics as the I0 were mounted onto the sample bar to assess the rate at which contamination grows onto new I0 meshes. One grid was mounted as it was received, and a second grid was cleaned via UVO for 90 minutes, although a few minutes passed between UVO cleaning and pumping down the sample bar. Scans were run at 0° polarization starting with scan IDs 96554 and 96555 for the as-received and UVO meshes, respectively. HOPG scans were run along along with the gold grid scans to generate carbon contamination. Unfortunately, it seems like the grids did not have sufficient electrical contact with the sample bar, and the signal was very weak. Across the scans performed, there was not a significant buildup of carbon contamination. The accompanying HOPG scans plots are also included mainly to highlight the evolution in the double-normalized signal. The double-normalized signal magnitudes across all scans show similar trends over time, suggesting that the evolution of the double-normalized signal likely is not related to any changes in carbon contamination on the gold grids.
Troubleshooting: Solid sample up-down motor
While beginning to calibrate positions for the solid sample bar fiducials, some errors occurred related to the up-down motor: one occurred prior to moving from y = 3.588 to y = 2.038 (UnknownStatusFailure: The status MoveStatus(done=False, pos=solid_sample_y, elapsed=0.6, success=False, settle_time=0.0) has failed.) and another occurred prior to moving from y = 3.588 to y = -187.34 (FailedStatus: MoveStatus(done=True, pos=solid_sample_y, elapsed=0.7, success=False, settle_time=0.0)). In both cases, no error message (e.g., amp fault, following error) was present in the EPICS-based motor control GUI, and retrying the motion command resulted in successful and complete motion. After this, it was attempted to move from y = -187.34 to y = 3.588, and a following error occurred in the middle of this motion. Moving from the failed point upward in one 1-mm increment resulted in smooth motion, but subsequent 1-mm increments were jerky and stopped prematurely. At this point, the motor velocity was reduced from 3 mm/s to 1 mm/s. Initial subsequent motions resulted in a few following errors, though these errors did not occur consistently during each command. A lingering issue that arose is that the motor successfully moved to the desired location, but then continued moving moving back and forth ~+/- 0.2 mm until the Kill button was pressed; see example videos of the motor control GUI and the handcrank motion. This issue occurs for upward and downward motions.
The motor controller was power cycled, which involved stopping and stopping the MC21 soft IOC and activating auto-disabling. Although following errors did not occur after this point, the continuous back-and-forth motion at the conclusion of a translation persisted. Notably, this issue did not occur at y = 345, but occurred at -10 < y < 50. The encoder shows a green light, suggesting that it is functioning properly. It is unclear at this point if there is a physical issue in the motor (e.g., weakened shaft) or a glitch in the signal processing.
This issue has occurred in the past, and retuning the motor has resolved the issue. The nature of the issue suggests that the motor is experiencing less resistance to movement. For the current issue, the motor was examined by connecting directly to the MCS8+ motor controller via ethernet (with the EPICS softIOC turned off) using the PEWin32Pro2 software. At y = 345, minimal issues were visible, which is consistent with the behavior noted above. At y = 0, it appeared that the motor was constantly in a negative-direction motion even though visually it appeared like the motor was oscillating in the negative and positive directions. After 40 ms of the Phytron stepper drives receiving no input from the PMAC, the current should change from the "run" current to the "stop" current. These are indicated by the yellow LED (run mode - receiving control pulses), green LED (stop mode / ready for input - turned on, but not receiving control pulses), and no light (motor killed). In the case of the y motor, the position starts to drift on the transition from run mode to stop/ready mode; after the position drifted far enough away, run mode was reactivated, which increases the current, causes a jump in position. This slipping occurred cyclically and continuously at some positions. This negative motion could be the result of something physically being loose such as a motor coupling; one possibility is that the motor shaft may be close to breaking, as has occurred a few times in the recent past, and the detuning is a symptom. In a standard NSLS2 PMAC controller, the stop current can be controlled directly. However, the MCS8 and Phytron drives are not as readily customizable. As a workaround, the deadband was adjusted such that the proportional gain would be reduced when the position arrives within the deadband; the lower proportional gain still produces oscillation, but with lower amplitude and lower frequency. Additionally, a kill command was explicitly added to the script so that after entering the deadband and reducing the proportional gain, the motion is completely stopped.
The physical workings of the motor were examined to troubleshoot the underlying issue, but no cause was found yet. To test the motor with its new tuning, various setpoints were set, and the motor was stopped either prematurely or after it arrived at its set point. Both the upward and downward motions were tested. In all cases, the motor appeared to behaving well, and these results suggested there are not physical issues with the motor. The position did not drift from a 345 mm setpoint over ~1 week, and it also did not drift significantly from a 0 mm setpoint over ~1 day.
Sample position calibration
The same sample bar from November 8, 2024 was used. The corresponding fiducial calibration scan IDs are 96313-96322. The bottom fiducials -90 and 90 degree positions needed manual calibration.
RE(find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.5667649999999753, -5.025, 0.47509999999999764, 3.7, -2.2216650000000016, -186.87915500000003, -2.016085000000004, -0.07313500000000062, 0.809829999999998, -1.7140900000000059], include_back=False)
This sample bar was removed, a few samples were added, and then the positions were calibrated again (image_bar scan ID is 96532). The above correct_bar positions were used as the initial guesses, and all fiducials were found automatically (scan IDs 96544-96553).
Troubleshooting: WAXS Greateyes CCD camera artifacts
This is a follow-up on the troubleshooting efforts from March 24, 2025 and April 3, 2025.
To deconvolute the effects of the sample features and the quadrant boundaries on the final image, a series of scans were performed on SBA-15 in which the WAXS camera was offset at a few different positions, starting with scan ID 96389 on proposal ID 318915. The Bluesky code used is pasted below to preview the order of scans. The camera is usually at a position of 2 by default for WAXS RSoXS scans. Scans also were run at -20 and -40 where it is pulled diagonally back and to the right-hand side (away from the beamline computers). At each detector position, a rough energy sweep was run from 100 to 1000 eV with 50 repeats; each energy sweep is its own scan ID. The polarization series is likely excessive for a sample like SBA-15 with no expected anisotropy, but it was run in case there is subtle anisotropy that coincides with quadrant boundaries.
## SBA-15 scans with WAXS camera moved to different positions
## To decouple sample features from camera quadrant boundaries
yield from load_configuration("WAXS")
## Load SBA-15 sample. This sample will stay in the same position throughout all scans.
yield from load_samp("SBA15")
add_current_position_as_sample(name="SBA15", sample_id="SBA15")
## Trying different polarizations in case SBA-15 has some anisotropy
#for polarization in [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180]:
for polarization in [0, 90, 180, 45, 135, 15, 30, 60, 75, 105, 120, 150, 165]:
yield from set_polarization(polarization)
## Iterate through different WAXS camera positions
## The WAXS camera comes in diagonally, so it would move both to the side and further from the sample.
for waxs_detector_position in [2, -20, -40]:
yield from bps.mv(Det_W, waxs_detector_position)
## Run an energy scan going from 100 eV to 1000 eV in 100 eV increments
## Running 50 repeat exposures at each energy at 0.1 s exposure time each.
energy_parameters = (100, 100, 1000)
yield from nbs_energy_scan(
*energy_parameters,
use_2d_detector=True,
dwell=0.1,
n_exposures=50, ## Was going to take 90 repeats, but then darks would be very infrequent
group_name="Assess WAXS camera quadrants",
sample="SBA15",
)A few attempts were made to perform scans in which a flat-field illumination could be achieved on the WAXS camera to assess the nature of the quadrant boundaries. In one attempt, a piece of aluminum foil was taped onto the sample bar. As X-rays hit the foil, fluorescence was generated and captured onto the WAXS camera. Energy sweeps were run at 800, 900, and 1000 eV with 50 repeats at each energy. Scan ID 96868 has 1 s exposures, and scan ID 96869 has 5 s exposures. After testing the positioning of the sample and camera (scan IDs 96516-96530), it was determined that having the sample bar angled at -70° (such that the sample faces the inboard side of the beamline) and having the detector fully retracted at -94 gave the largest coverage of illumination. Still, some parts of the camera sensor were blocked, possibly by the uneven surface of the sample. The illumination is likely not truly flat, as there might be an intensity gradient from the outboard to inboard.
July 16, 2025
PGM repairs
Following the cooling line leak that occurred in the PGM on June 21, 2025, FMB Berlin has started repairs.
After the PGM chamber was vented and opened, it was confirmed that all air guard bellows do not have leaks. The corresponding air guard for the hose that broke expanded significantly (without breaking) as a result of the water pressure. This bellows will be replaced along with the plastic hose.
All plastic hoses were removed, as the plan is to replace all hoses regardless of whether they are broken. Another hose was identified to be close to breaking. The remaining hoses will be assessed closer to decide whether they are worth keeping as spares.
July 11, 2025
Beam alignment
The ring current is planned to increase from 400 mA to 500 mA on July 14. Studies time was provided on July 11 to align beamline optics. As the PGM is still down, alignment and beam characterization was only performed up to FS6.
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.52 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.2 | -0.6 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -5 | -5 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 5 | 5 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -5 | -5 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.5 | 1.3 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.57 | 0.57 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
1D parameter sweeps were performed on the M1 parameters (roll, yaw, x, and pitch) scan IDs 95955-96279.
June 21, 2025
PGM leak
It was noted today that a water-cooling line inside the PGM broke, and water flooded out the inlet and outlet air guards. This caused flooding around SST beamlines and also neighboring QAS hutch.
The cooling line assembly consists of an inner plastic tube, a fiber glass sheath, and an outer metal air-guard tube that is welded onto flanges. It is unclear why the inner tube broke, but a similar issue had occurred at the CSX beamline; a possible reason could be wear and tear. Both air guards leaked because the cooling line bridges across all gratings in the chamber, and the air guarding is interconnected across this assembly. Due to the current design, the water leak was not channeled to a drain, so water flooded over out-of-vacuum hardware and over the floor. The 7-ID leak system also did not detect the leak (and pressure change) and shut off valves; instead, the QAS leak detection system is what got triggered first.
In the plot below, the water flow dropped and photon shutter 4 (PSH4, at the downstream end of the FOE) closed at ~3:30 pm. As a result, the gratings did not overheat. The front end shutter did not close likely because PSH4 already was closed. The blip in the water flow (purple) is about when the FOE alarm tripped at ~4:30 pm and likely when water flow stopped. Thus, water was likely flooding out onto the experimental floor for ~1 h; ~200 gallons of water was lost.
Until the repair is complete, the beamline will be down. A full repair likely will require assistance from FMB Berlin, as the cooling mechanism should be set first, and then the optics can be aligned more finely. In the meantime, motor movements can be tested after the assembly is fully dried; as water flooded onto some hardware (worm gears, encoders, etc.), it should be checked if the hardware still works.
In the future, the air guard can design can be improved such that the water is channeled toward a drain so that water does not damage hardware and flood the floors. Also, interlocks and alarms can be set using Phoebus and Bluesky such that shutters and gate valves are closed, water flow is shut off, alarms are sounded, and notifications are sent by email immediately when a drop in water pressure is detected. Physical interlocks also can be considered.
June 7, 2025
The same beam alignment as June 4, 2025 was used.
Sample and energy calibration
A sample bar from the same batch as the one used November 8, 2024 was used. The corresponding fiducial calibration scan IDs are 95525-95534. All fiducials were found automatically without the need for manual location.
RE(find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.259079999999983, -2.6829350000000005, 1.1770149999999973, 1.1515349999999955, -2.6720450000000042, -187.33704500000005, -1.9353150000000028, 0.22979499999999575, 0.9090649999999982, -1.4094550000000012], include_back=False)
The same energy calibration as June 4, 2025 was used.
June 4, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
400 mA ring current |
Final values at by the end of beam time,
400 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.52 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.2 | -0.2 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -5 | -5 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 5 | 5 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -5 | -5 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.5 | 1.3 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.57 | 0.57 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.2 | 24.2 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 17.5 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.78 | 7.8 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
Sample and energy calibration
A sample bar from the same batch as the one used November 8, 2024 was used. The corresponding fiducial calibration scan IDs are 95295-95304. All fiducials were found automatically without the need for manual location.
RE(find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.283379999999994, -2.6165599999999998, 1.1005449999999968, 1.2415850000000006, -2.4317800000000034, -187.10289, -1.959805000000003, 0.005189999999998918, 0.7818599999999947, -1.3906750000000017], include_back=False)
Energy calibration scan IDs are 95313-95327. X offsets for mirror and grating positions are 2.665e-02 and 2.190e-02 respectively. A reference HOPG scan was taken with scan ID = 95328 at 0° polarization as well as scan ID = 95329 at 90° polarization (with sample bar rotated 70° to get maximum beam footprint) and showed good resolution.
May 27, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
400 mA ring current |
Final values at by the end of beam time,
400 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.52 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.05 | -0.2 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -2.45 | -5 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 2.86 | 5 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -3.45 | -5 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.5 | 1.3 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.57 | 0.57 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.2 | 24.2 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.87 | 7.8 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
It is noted that the center of the beam on FS6 actually was found at a FE:C07A-OP{Slt:34-Ax:Y}center value of -0.75. However, it was changed to -0.2 because -0.75 placed the beam too low on the RSoXS station, and the beam would be too far off-center from the WAXS camera.
Some time lapses were captured on the RSoXS fluorescence screen that need to be processed. During June 2-4, the soft beam was transferred over to the tender SST-2 beamline, and there were suspicions of beam drift upstream of the monochromater. During initial alignment, a sudden loss of signal was experienced after which the signal was regained after realigning the beam. Later on, XPS scans were conducted across various energies; at each energy, a gold reference scan was taken before and after the sample scans. At 1400 eV, for example, there was a 0.1 eV shift and a significant drop in intensity between the initial and final reference scans even though beam alignment settings were not changed. A closer look at the RSoXS fluorescence screen timelapse can help confirm if something similar is happening with the soft bema in the SST-1 beamline.
Troubleshooting: I0 and TEY signals
The PXIe crate had been rebooted on May 20, 2025. This crate has the ADC cards used to read the RSoXS I0 and TEY signals (PV names XF:07ID-BI[ADC:2-Ch:0]Volt and XF:07ID-BI[ADC:2-Ch:2]Volt, respectively). When the dark signals are read out via EPICS (e.g., caget XF:07ID-BI[ADC:2-Ch:0]Volt for reading out a single data point, and camonitor XF:07ID-BI[ADC:2-Ch:0]Volt for streaming multiple data points after which ctrl+c can be used stop monitoring), the values were non-zero and looked reasonable (e.g., -0.000200146 for XF:07ID-BI[ADC:2-Ch:0]Volt and 0.117337 for XF:07ID-BI[ADC:2-Ch:2]Volt).
However, these signals do not show any response to the presence of beam; i.e., the order of magnitude does not change. After some initial troubleshooting for the I0 signal, it was found that the SRS570 preamplifier upstream of the ADC card receives the correct signals, outputs voltage, and shows responsiveness to beam vs. no beam. The input was tested by saturating the SRS570 and checking for the overload error to show when beam was hitting the I0 mesh. The output of the preamplifier also showed a 0.01 V reading with beam hitting the mesh vs. 0 V with no beam on the mesh. For reference, these tests were run with the following SRS570 settings (the unit that was originally used for I0): no bias voltage, filter type = 6 dB high pass, gain mode = low noise, input offset = positive 1x1 pA, filter frequency = 100 Hz, sensitivity = 5x10 nA/V, power = line. Additionally, a Keithley picoameter also was connected directly to the I0 cable and showed ~14 nA readings. TEY signal was verified by connecting to an I400 electrometer. For reference, the SRS570 unit that originally was processing the TEY signal used the same settings as the one that originally processed the I0 signal except filter frequency = 30 Hz and sensitivity = 1x1 nA/V.
It is noted that the XF:07ID-BI[ADC:2-Ch:7]Volt PV (originally used for the DM7 photodiode signal), which comes from the same ADC card, but a different channel, shows full functioning and responsiveness to the beam. For now, the I0 signal is being collected on a different SRS570 that connects to the XF:07ID-BI[ADC:2-Ch:7]Volt PV ADC channel. For reference, the SRS570 settings for this unit are: no bias voltage, filter type = None, gain mode = low noise, input offset = positive 1x1 pA, filter frequency = none, sensitivity = 1x10 nA/V, power = line. The TEY signal is being collected on an I400 (channel XF:07ID-ES1{Slt1:I400-1}:IC2_MON), but the TEY signal is very noisy. Starting June 4, the TEY signal started being collected from XF:07ID-BI[ADC:2-Ch:7]Volt PV, and the I0 signal was taken from an I400 (channel XF:07ID-ES1{Slt1:I400-1}:IC1_MON).
Further testing was performed to check that the hardware downstream of the SRS570 pre-amplifier works well. A Keithley 487 picoammeter was used to generate voltages from 0 to 5 V, and this source was connected to the cables that originally were connected to the outputs of the SRS570 pre-amplifiers on one end and the XF:07ID-BI[ADC:2-Ch:0]Volt and XF:07ID-BI[ADC:2-Ch:2]Volt ADC channels on the other end. As shown in the plot below, the channels showed appropriate responses to the voltage source, suggesting that the cables, ADC card, downstream electronics, and channel archiver are working.
At this point, all hardware appears to be functioning properly, and it is unclear why there was an issue reading signals initially. During the next beam time, the detectors will be connected back to these original channels and tested further.
It is noted that a possibly similar issue was encountered with one of the SRS570 units at the HAXPES end station on July 22, 2025. Restarting the soft IOC seemed to get it working again.
Sample and energy calibration
The same sample bar from November 8, 2024 was used. The corresponding scan IDs are 94869-94878. The bar may have twisted along the up-down axis, so the -90° and 90° fiducial positions were not found automatically and needed manual searching.
RE(find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([], include_back=False)
Energy calibration scan IDs are 94884-94898. M2 positions, PGM positions, energies, and orders were [-3.0810908473669087, -2.9527227613284097, -2.8433797861814085, -2.7489246999049115, -2.666397434513911], [-3.646464151380499, -3.542814120803996, -3.4563431295534954, -3.383485098274001, -3.320601402266], [291.65, 291.65, 291.65, 291.65, 291.65], [1, 1, 1, 1, 1], respectively. Angle offsets for mirror and grating positions are -1.472e-02 and -1.087e-02 respectively. The data for these scans are extremely noisy, but they do show the expected features as well as expected flux trends with CFF (i.e., CFF = 1.45 has the best energy resolution, and thus, the largest peak intensity). A reference HOPG scan was taken with scan ID = 94899-94903; these scans showed okay resolution on their own, but it is very poor in comparison to scan ID 84990 from 2024. The data was extremely noisy. This may be due to the I400 electrometer being used to collect TEY data.
Reference energy scans
Open-beam energy scans were performed at a variety of edges and polarizations with scan IDs 94912-95292. It is noted that there was a brief interruption to beam flux because Precision Slit C gap accidentally got changed during a scan at ~8:50 am ET on May 29, 2025. On the same day, the beam also went into decay mode from ~12-1 pm ET, after which the full 400 mA current was restored.
The photodiode signals are generally noisy. Given that the I0 signals do not show similar noise, the photodiode noise is likely an issue with the cabling for the WAXS photodiode. On the other hand, the I0 signal shows a slight increase over time, whereas the photodiode signal is relatively steady. This might suggest that contamination is building up on the I0 mesh over this time period. The I0 gold mesh position and the slits1 positions were not changed from prior beam times, but possibly the changes in beam alignment might impact what local regions of the I0 mesh get exposed; fresh locations might be receiving exposure that were not exposed to beam before. Otherwise, however, the normalized signals do not vary drastically.
The energy dependence of the beam flux is similar to that seen during the April 3, 2025 beam time and follows the expected flux calculated during November 2023. In fact, the fluorine edge scans show an increasing trend in signal unlike what was seen during the April 3, 2025 and January 30, 2025 beam times; this is expected based on what was seen May 2023 and November 2023. Given the poor energy resolutions in the HOPG scans above, it is surprising that the energy calibration was performed successfully, scans are as smooth as they are across all edges, and scans show similar features as the past.
May 20, 2025
On May 2, 2025, the conda environment was updated from /nsls2/conda/envs/2024-2.3-py311-tiled to /nsls2/conda/envs/2025-2.0-py311-tiled. Additionally, the profile_collection and rsoxs codebases (but not rsoxs_workflow) were migrated from the NSLS-II SST GitHub to the NSLS2 GitHub. Further developments are detailed below.
Beamline computer setup
Changes were finalized on the beamline control computer (xf07id-ws24) by activating the new conda environment and pip installing the necessary packages (the nbs_* packages, sst_base, and rsoxs); for each package, first navigate to the local repository using cd, and then install by running pip install --no-build-isolation -e .. The packages need to be installed because they use entrypoints, and they cannot be gathered into a single file path (e.g., PYTHONPATH).
It was attempted to pip install packages into /nsls2/data/sst/rsoxs/shared/config/bluesky_overlays/2025-2.0-py311-tiled, but permission errors were encountered.
The ~/.bashrc document (accessible by running emacs ~/.bashrc) was used locally to override some variables (e.g., BS_PYTHONUSERBASE_PATH) that were incorrectly set by ansible, which is used to centrally manage Linux computers by pushing config files down onto them. The ansible script /etc/profile.d/bluesky.sh on a central server had some errors, which were corrected locally, as the ~/.bashrc is the last set of commands that gets loaded while opening a terminal. The ~/.bashrc document also defines commands such as bsui, bsqs, and bsgui.
Note that although running bsui in a terminal uses some default PYTHONPATH and PYTHONUSERBASE variable, these file paths are not actually used.
GitHub permissions
The original intention was for all beamline staff to join the SST team and have maintainer role. However, there are now a limited number of seats for how many users can have maintainer/admin access to the repository. The beamline scientist, along with several external collaborators, lost this access. Ideally, this loss of access is temporary, and access will be restored with the help of BNL's IT department. Until then, only a single SST staff member (not the beamline scientist) a few DSSI staff can approve pull requests. The beamline workstation also has deploy keys to push changes to GitHub. All other contributors will need to create forks and get changes approved by the aforementioned staff. It is planned to migrate the rsoxs_workflow codebase to the NSLS2 GitHub only after the above issues are resolved, and working access is confirmed.
For ease of syncing codebase changes, the beamline computer could point to a fork of a repository temporarily instead of the main NSLS2 repository. First, navigate to the local repository using cd.The link to a repository can be removed by running git remote remove origin. A new link could be created by running, e.g., git remote add fork_usnistgov git@github.com:usnistgov/sst-rsoxs-profile-collection.git (or more generically, git add <name for remote link> <remote link>); a list of remote options can be viewed by running git remote -v. Then, specify which repository/branch to pull by running, e.g., git checkout -b my_branch fork_usnistgov/my_branch. Future updates to this fork on GitHub web can be synced by running, e.g., git fetch fork_usnistgov, and then updates can be pulled by running git pull.
As of June 10, 2024, the beamline scientist has been added to the SST team, but cannot add external collaborators. Thus, the usnistgov fork will be continued for use at the beamline computer and collaborating with the NIST team.
May 5, 2025
Greateyes WAXS camera updates
Updates to the Greateyes WAXS camera dimensions and chunking are to be rolled out. See the Github issues below for details.
Feature request: RSoXS users email list
In light of recent retroactive metadata change initiatives, it would be good to have an email list that includes all RSoXS users to send announcements for roll-outs that may disrupt data access. An automated mechanism to gather email addresses from RSoXS proposals using a PASS API would be ideal, but PASS-api is on the campus network (not the science network), which limits access. Currently, proposals can be queried to nsls2api and includes email addresses. However, RSoXS does not have its own proposal directory. Instead, there is a directory for SST proposals, /nsls2/data/sst/proposals, that is shared with SST1 RSoXS, SST1 NEXAFS, and SST2 HAXPES. An alternative is to generate an email list manually.
April 17, 2025
BNL blocking access to Anaconda
It was announced that BNL will block access to Anaconda on all beamline computers due to Anaconda's recent licensing changes. BNL recommends switching to conda-forge and miniforge. Another options for users reducing data is JupyterLab Desktop. Progress on Python setup instructions can be found here: https://github.com/NSLS-II-SST/rsoxs_workflow/issues/46.
April 3, 2025
Beam alignment
The initial values were those that were aligned on March 27, 2025.
| Description | Process variable name | Initial values at the start of beam time,
400 mA ring current |
Final values at by the end of beam time,
400 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.52 | 0.52 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.05 | -0.05 |
| FOE pink beam slits 01, outboard | XF:07IDA-OP{Slt:01-Ax:O}Mtr.RBV | 3 | 3 |
| FOE pink beam slits 01, inboard | XF:07IDA-OP{Slt:01-Ax:I}Mtr.RBV | -2.45 | -2.45 |
| FOE pink beam slits 01, top | XF:07IDA-OP{Slt:01-Ax:T}Mtr.RBV | 2.86 | 2.86 |
| FOE pink beam slits 01, bottom | XF:07IDA-OP{Slt:01-Ax:B}Mtr.RBV | -3.45 | -3.45 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | 1 | 1 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.5 | 0.5 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.05 | 24.2 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.78 | 7.78 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
The FOE pink beam slits are used to remove some low-intensity beam "reflections" that have been seen during past beam alignment efforts. These may be artifacts of a bending magnet, but will need to be investigated further.
Sample and energy calibration
The same initial guesses as March 24, 2025 were used for searching fiducials, even though this was a different bar of the same batch. The corresponding scan IDs are 94020-94029, and all fiducials were located automatically.
RE(fly_find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.6303299999999865, -2.403090000000006, 0.8659449999999964, 0.8941949999999963, -2.4322350000000057, -186.94343500000002, -2.147725000000001, 0.21978499999999457, 0.5957749999999962, -1.440135000000005], include_back=False)
Energy calibration scan IDs are 94030-94044. A reference HOPG scan was taken with scan ID = 94046 and showed good resolution on its own, though it still seems less sharp than scan ID 84990 from 2024.
Troubleshooting: fast shutter malfunction
At the start of the beam time, the fast shutter seemed to be stuck almost all the way open. This issue was first noticed when the fast shutter was moved into the beam path, and signal was seen on the WAXS beamstop (with the CCD camera retracted) even though the EPICS readout of the fast shutter was closed. It was confirmed that the positions of the shutter and slits were correct by homing them, so misalignment was likely not the issue. The amplifier for the fast shutter (dsm linear piezoelectric amplifier, model: VF-500 XR5, serial number: E18-01499) was showing the AUX light as on even when EPICS readout of the fast shutter was closed and also when the input BNC cable was disconnected; in both of these cases, the AUX light should turn off. Power cycling the amplifier did not resolve the issue. However, when the power is switched off, the shutter closes.
Various troubleshooting was performed, but finally, the fast shutter began working after switching off the amplifier power for several hours before restarting as well as disconnecting and reconnecting the output from the amplifier into the shutter. It is unknown which of these actions resolved the issue or why. Several NEXAFS measurements were performed with the WAXS CCD camera retracted to check that the shutter reliably opened and closed (i.e., WAXS beamstop and sample current signal intensity decreased to 0 at the end of each scan). Then, spiral measurements were performed using the WAXS CCD camera while closely monitoring beamline operations to ensure the shutter does not fail open and damage the camera. Similarly, scattering energy scans were performed with close monitoring. For reference, the same issue had occurred in ~2017 when the RSoXS station was being commissioned; the amplifier had been completely replaced, as it was under warranty at that time. Moving forward, it is recommended to have a backup amplifier unit, as a malfunctioning shutter is a single point of failure for the beamline. The vendor of the amplifier also has been contacted for advice on further characterization of the issue and prevention of a failure. It is possible the initial failure was due to overheating and that a prolonged power-off cooled down the system.
While troubleshooting the output cable from the amplifier into the shutter, an error was identified in the manual. The shutter is connected to a 15 pin female cable, and the manual stated that pins 1, 2 carry the voltage and pins 3, 4 are ground; however, pins 9, 10 are actually ground. It was also noted that whenever the amplifier was switched on or off, there would be a large spike in voltage before it steadied at a low/moderate value.
Some brainstormed workarounds are noted here for reference. The workaround that was used temporarily was to move the fast shutter vertical translation to mechanically block the beam; the beam was blocked at a position of 25 but unimpeded at a position of 26. The use of photon shutter 10 (PSH10) was discussed for NEXAFS use. However, this avenue was not explored further due to a concern that PSH10 was not designed for opening and closing every few minutes, and that this sudden increased use could damage the hardware. This concern would be even more relevant for RSoXS scans with the WAXS CCD camera in which the shutter would need to be opened/closed every ~5-10 seconds; additionally, PSH10 would not be able to close fast enough before the CCD starts reading out images. Although the shutter vertical translation also could have experienced some wear-and-tear, the distance moved was significantly smaller than that moved by PSH10, which can reduce the wear in comparison to the pneumatic actuator used for PSH10. The option of temporarily replacing the amplifier with an E-625 PZT servo controller also was considered, but it was unclear if the correct voltages could be matched.
Troubleshooting: Uneven quadrants in WAXS Greateyes CCD camera
Following up on the troubleshooting efforts from March 24, 2025, a series of test scans were collected using a reference sample with significant scattering anisotropy. Scan IDs 94431-94434 are carbon-edge RSoXS scans with a single image per energy read out using a single ADC (quadrant E) at 0, 45, 90, and 135 degrees polarization, respectively. Scan IDs 94435-94439 are identical scans, except the images were read out using all 4 ADC channels.
One difference between the 4-ADC readout vs. 1-ADC readout cases was that the 4-ADC readout images appears to be missing a ~2-pixel-tall strip of the image at the horizontal junction between quadrants. It was noted that the 4-ADC images had dimensions of 1024 x 1026 (after 4x4 binning), whereas the 1-ADC images had dimensions of 1024 x 1028. However, the 1026 vs. 1028 refer to the inboard-outboard (horizontal) dimension, whereas the "missing pixels" are in the vertical direction. [TODO: add plots]
Anisotropy ratios were calculated for all cases. The anisotropy ratio profiles showed similar shapes for the 4-ADC readout vs. 1-ADC readout cases. However, the 4-ADC case showed scaled down anisotropy values relative to the 1-ADC case even though the overall intensity was normalized in the process of calculating anisotropy ratios. Other factors to investigate include dark subtraction, noise differences, and/or background differences. [TODO: add plots]
The camera artifacts are being discussed with engineers from Greateyes. It was found that the integrated non-linearity of upper right-hand quadrant is different than the other 3 quadrants, especially in < 20000 ADU range. Further discussion is in progress to understand if this non-linearity may be related to any of the other artifacts noted here. [TODO: add plots]
Although single-exposure scans did not reveal obvious intensity offsets across different quadrants, scans with repeat exposures on weakly anisotropic samples brought out these differences more clearly. Offsets were found across all quadrants, not just the upper right-hand quadrant. There was also "cusps" within each quadrant near the boundaries.
EPU error
While starting a scan on April 5, 2024 in which the EPU was to be moved to 0° polarization, there was an error in Ax:BO and Ax:BI. Calling the controls room and running a quick reset on the EPU seems to have resolved the error. The scans are being monitored to see if the error occurs again.
Beam decay due to failed Klystron 1
Since 5:08 pm on April 5, 2025, the beam had been in decay mode due to Klystron 1 failing in the LINAC. Unlike regular beam dumps, the beam decay did not automatically close the front-end shutters, and scans continued to run until ~8 am on April 6, 2025. Any data taken during this period should be interpreted with caution.
Open-beam energy scans
Open-beam scans were performed at various times throughout the beam time. There are certain scans in which the photodiode signal becomes noisy. This is likely an issue with the cabling for the WAXS photodiode.
The energy dependence of the beam flux is roughly similar to that resported during the January 30, 2025 beam time. One main difference is that the fluorine K-edge signals have noticably higher flux than the oxygen K-edge, which is closer to the expected flux.
In some cases, the normalized signals did not all collapse onto a single master curve.
It is also noted that during acquisition, the beginning energy values stored in the baseline datastream did not accurately represent the actual first energy value in the scan. This bug may be present in other data too, but was noticed first here.
March 24, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
400 mA ring current |
Final values at by the end of beam time,
400 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.2 | 0.2 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.6 | -0.6 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.48 | -0.55 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.532 | 0.45 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.05 | 21.8 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.78 | 7.91 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
Sample position and energy calibration
The same sample bar from November 8, 2024 was used. The corresponding scan IDs are 93893-93902. The bar may have twisted along the up-down axis, so the -90° and 90° fiducial positions were not found automatically and needed manual searching.
RE(fly_find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.492869999999982, -4.8, 0.7703349999999958, 3.9, -1.966345000000004, -186.91449000000006, -1.8170800000000042, 0.21111999999999398, 0.9979549999999975, -1.2376600000000053], include_back=False)
Energy calibration scan IDs are 93903-93917. X offsets for mirror and grating positions are 2.198e-02 and 2.307e-02 respectively. A reference HOPG scan was taken with scan ID = 93918 and showed good resolution on its own, though it still seems less sharp than scan ID 84990 from 2024.
Troubleshooting: Uneven quadrants in WAXS Greateyes CCD camera
This is a follow-up investigation for the issue that occurred during the February 21, 2025 beam time.
Characterization of detector images
First, the issue was recreated. The camera power had not been switched off between February 21, 2025 and this point. An SBA-15 sample was measured at 560 eV, 0° polarization for all tests so that the fluorescent background would aid in noticing differences between the WAXS camera sensor quadrants. It was found that the the differences between the upper right quadrant and the other quadrants became less apparent as exposure time increased, possibly because of the increasing fluorescent background with exposure time; i.e., on a log-scale image, small differences between quadrants are more apparent when the percent difference is larger. However, the opposite trend seems to appear for an SiN blank window at the same exposure series, which has less fluorescence.
[TODO: plot in linear scale to compare with above log scale plots to get a better sense of difference between pixel values.]
Although the issue is visually apparent in the images, it has been challenging to characterize the intensity difference by taking a line cut across the the quadrants. A step profile is likely lost in the noise of the image. Similar challenges were present when trying to average narrower regions across the top of the image.
There could be a few phenomena contributing to the uneven quadrants:
- The analog-digital converter (ADC) for the upper right-hand quadrant may have a different gain (different signal per electron) than the other quadrants. This cause may be unlikely, as on a log scale plot like those above, the quadrant differences should appear the same at all exposure times due to all signals being offset by the same ratio; that is not the case in the above plots.
- A level offset in which there is a drift in the upper right-and quadrant. This phenomenon should be most apparent at low signals, and is common with CCD sensors; it could be normalized by using a blanked out part of the image. Obtaining a functional form of the offset vs. signal (exposure) and offset vs. time could provide further information; it would be useful to know if the difference between quadrants is a constant or a gradient. However, the contrast across quadrants is worst at the highest exposure times instead of the lowest exposure times in the case of the SiN blank window images, suggesting that a level offset may not be the cause of this issue.
- A noise floor difference across the different ADCs. This phenomenon also should be most apparent at low signals, but this is not the case for the SiN blank window images. However, this may be a likely cause given that a step difference was lost in the noise of the line cuts. In the image below, the original 1024 x 1024 pixels images from the SiN exposure series were re-binned to form 256 x 256 pixels images. These re-binned images do not show a distinct upper right-hand quadrant, suggesting that the issue with the original image is primarily a difference in noise.
All three of these phenomena may be affected by uneven heating across the ADCs. The manufacturers suggested that the sensor itself has different conversion gains for each quadrant, and the readout electronics also use four separate amplifiers and ADCs; a background subtraction might eliminate/reduce the offset differences from the latter but not the conversion gain, and a flat field correction might be needed for the conversion gain.
Adjusting camera settings
Prior to switching off the camera, various adjustments in the camera settings were tested. These settings were accessed from Phoebus by navigating to "SST Launch" --> "RSoXS Station" --> "GreatEyesCCD" --> "GE2 WAXS" --> "Expert (AD detail)" --> "ADgreatEyesCCD". Images captured at 1 s exposure time using the readout mechanism from a single quadrant were uniform, suggesting that the the issue with the upper right quadrant might be related to the readout mechanism rather than the sensor pixels. Additionally, reading out from quadrant E appears to have the least artifacts (e.g., streaks). So far, this is the only solution that preserves the camera's low noise and high dynamic range, but the readout times are 4 times larger than those in which all 4 quadrants are used for reading out. It is also noted that when using a single quadrant to read out images, the image size increases to 1024x1028 pixels (from the original 1024x1026 pixels). For future reference, a dark image should be taken before each single-ADC image.
In theory, readouts can be performed using two quadrants, but this option ran into several challenges. Images only could be captured using the "2 Amps, 2 Reg" readout with quadrants E and F. The remaining quadrant pairs resulted in the camera server being disconnected before an image was captured. Even while using quadrants E and F, the images were non-uniform, and the non-uniformity was inconsistent over several images.
Finally, it was attempted to adjust the capacity and gain while reading out from all 4 quadrants. The camera uses standard capacity (low noise) by default. Changing to extended capacity made the quadrants appear uniform (scan ID 93963), but may be the result of a larger noise floor. It was attempted to change the gain from low to standard, but this change never got updated in the camera software; in any case, it would be undesirable to increase the gain, as it would reduce the camera's dynamic range. Minor adjustments to the gain using the "ADBase" settings did not resolve the quadrant issue; the default gain is 1, and scans were performed at 0.9 (scan ID 93969) and 1.2 (scan ID 93970) gain. Further setting options may be available through the greateyesVision software provided by the camera vendor, but it is unclear how the fast shutter would be synchronized with the camera; this route may be explored further with assistance from Greateyes.
After running the tests above, the camera power was switched off for 8 minutes, as this issue had been temporarily resolved in the past by powering off the camera. However, the upper right quadrant issue was still present after powering on the camera and allowing it to cool down to -78 °C. The same issue is present after powering off the camera for 3.5 h. This was not tested further, as frequent power cycling and warming of the camera sensor could cause other damage.
Possible workarounds and future troubleshooting
As shown above, reading out from a single amplifier instead of 4 amplifiers ensures a homogeneous image to some extent, but increases readout times by 4x. This longer readout time also increases the risk of the ADC heating significantly such that a gradient may form across the image (i.e., pixels read out later are warmer than those read out earlier). One way to counteract the longer readout time is to increase binning. Larger binning would reduce spatial resolution and the dynamic range, the latter of which could be addressed with repeat exposures.
Another separate workaround for anisotropic samples is to use 45°/135° beam polarization instead of the usual 0°/90°; this workaround needs to be tested using anisotropic samples, especially because the beam center is not exactly at the center of the CCD sensor. It also might be helpful to compare anisotropy calculations for an oriented sample using 4 vs. 1 amplifiers; if only a noise difference is present, it might be fine to use the 4-amplifier readout for oriented samples. Another method to test is to use a circular polarization image as the normalization basis for the linear polarization images.
Further solutions may arise either through hardware assistance from Greateyes or methods to correct the background after a scan is completed.
It may be helpful to monitor the temperature of different components, which can be obtained by using ssh to access more camera readout parameters. There appears to be a gradient within the upper right-hand quadrant which may be due to the ADC heating while reading out, though a gradient due to heating should be linear, not diagonal. Examining the dark images may provide further insight.
Further quantitative analysis of the images also will help narrow down the issue. The read noise of each pixel should be calculated. It also can be attempted to construct a photon transfer curve by using a dim LED light to produce a flat-field illumination across the sensor for a series of exposure times.
March 19, 2025
March 13, 2025
An additional datastream was added to scans which provides the timestamp for each individual image. Prior to this, scans with multiple images per energy only showed a single timestamp. Details are in a Jira ticket: https://jira.nsls2.bnl.gov/browse/SXSS-589.
Around the same time, another issue was fixed in which the "timestamps" datastream was not accessible through Tiled: https://jira.nsls2.bnl.gov/browse/SXSS-590
March 8, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
400 mA ring current |
Final values at by the end of beam time,
400 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.3 | 0.2 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.6 | -0.6 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.55 | -0.55 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.45 | 0.45 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 22.1 | 21.8 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.92 | 7.91 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
The portion of the beam visible on FS1 appeared normal, but the beam visible on FS6 looked different from prior beam alignments. It seemed like the beam was cut off on the bottom right-hand side, and the top left-hand side had a rounded corner. Also, a combination of mirror 1 parameters was not found in which gaps from the beam would be removed. For now, closing the front-end slits allowed a clean portion of the beam to be selected, but the beam profile may need further investigation. An image of the beam was not found on FS7, and it is unclear if the camera does not work.
Beam centering was performed based on scan ID 93236, which is a scattering image of SBA-15 at 291.65 eV, 0° polarization. The beamstop was masked with a quadrilateral polygon with the following vertices in the x (inboard-outboard) and y (up/down) coordinates: ["{\"x\":{\"0\":535.158,\"1\":510.461,\"2\":-0.476,\"3\":-0.476,\"4\":535.158},\"y\":{\"0\":463.025,\"1\":431.255,\"2\":848.658,\"3\":925.668,\"4\":463.025}}"]. The beam center was found to be [458, 504] (pixels in the inboard-outboard and up-down directions, respectively), and sample-to-detector distance was 31.005248154973497 mm.
Energy calibration
Energy calibration scan IDs are #-93126. X offsets for mirror and grating positions are -3.191e-2 and 3.067e-2 respectively. A reference HOPG scan was taken with scan ID = 93127 and showed good resolution.
As an additional validation test, scan ID 993237 was performed in which SBA-15 scattering images were captured over a large energy range at 0° polarization. The actual energies were extracted from the Bragg peak radius based on the sample-to-detector distance that was calculated above at 291.65 eV. The results show a systematically increasing deviation from the setpoint energy as the setpoint energy gets further from 291.65 eV. However, it needs to be tested if this deviation is due to an issue with the energy calibration or possibly an inaccurate reference length scale of SBA-15 (8 nm). This can be tested by taking multiple images at 291.65 eV with the sample z motor at different positions (if possible) such that the difference between sample-to-detector positions is known based off the motor positions. Another similar approach is to move the WAXS camera to different positions; however, the WAXS camera moves in a diagnal direction, so it is not immediately known how the WAXS camera motor positions corresond to changes in sample-to-detector distance, but likely is solvable with a set of multiple WAXS camera positions.
Sample position calibration
February 26, 2025
Data security migration
The primary purpose of this beam time was to upgrade the beamline codebase and environment to comply with NSLS II's data security developments. RSoXS beam time routines were tested after the migration and remain functional as before. The main changes that will be experienced on the user end are listed below.
- Users must authenticate into their proposal at the start of beam time:
sync-experiment -b SST1 -e rsoxs -p 316367, in which is 316367 is an example proposal number. - Spreadsheets will be temporarily stored in
nsls2/data/sst/shared/scratch/instead ofnsls2/data/sst/legacy/RSoXS/bars/barsduring beam times. The spreadsheets must be moved over into the proposal directory at the end of the beam time, where they will be stored longer-term. The updated temporary directory is the result of the planned elimination of the legacy directory. This temporary storage is required for spreadsheets because it is currently impossible for the beamline computer to have access to the proposal directories, so spreadsheets cannot be retrieved from the proposal directory to import into Bluesky.
Electrometer troubleshooting
This is a follow up to the I400 electrometer issues reported in December 2024, which were suspected to be the result of an inappropriate electrical grounding setup. After further discussion with other SST staff, it was suggested that electrical grounding should not impact the cross-talk that occurs among channels on the same I400 unit. The cross-talk issues in I400 electrometers seems to be a universal issue across several SST measurement stations.
It was suggested that a single-channel device could eliminate cross-talk possibilities and allow more fine tuning of the gain and other parameters for each individual signal. Preliminary tests are being performed by reading the RSoXS I0 and sample TEY currents through an SRS570 preamplifier + ACD card that is mounted onto a PXIE crate. Based on some comparison scans, there was noticeable improvement in the I0 noise. The sample TEY did not show noticeable differences on an HOPG energy scan, but this signal will be tested for other TEY samples.
Currently, other SST measurement stations already use the SRS570 + ACD combination, and longer-term plans are to replace the ACD card and PXIE crate with a PandAbox to support fly scanning developments. It might make sense migrate RSoXS signal measurements to this system to make sure of collective beamline knowledge and troubleshooting efforts. An alternative option is to consider a different multi-channel device such as the TetrAMM and investigate if cross-talk would be a similar concern for these devices.
February 21, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.2 | 0.2 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.51 | -0.6 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | 0 | -0.55 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.575 | 0.45 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.1 | 22.1 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.88 | 7.93 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
The alignment was set as it was due to past successes. The initial alignment settings that were used at NEXAFS will be tested further at RSoXS during a future commissioning beam time.
Sample position and energy calibration
A new sample bar manufactured in the same batch as the bar used on November 8, 2024 was used. The same initial guesses used on February 13, 2025 were used for this bar, and the automated scans found good fits, although the two top side fiducials showed weak signal. The corresponding scan IDs are 92731-92740.
RE(find_fiducials(f2=[-2.7923677459465375, 1.0046238327463828, 1.0805282896897526, -2.7630402458556915], y2=3.2577556976279864, f1=[-2.2560237681427995, -0.01548851211297432, 0.6519429066640681, -1.544682725471244],y1=-187.36415907091057))
correct_bar([3.3653449999999907, -2.7474200000000053, 0.3959199999999967, 0.9911849999999944, -2.1671949999999995, -186.83829500000002, -2.0489750000000058, -0.009240000000005466, 0.5485449999999972, -1.4234650000000002], include_back=False)
Energy calibration scan IDs are 92741-92755. X offsets for mirror and grating positions are 1.297e-2 and 1.326e-2 respectively. A reference HOPG scan was taken with scan ID = 92756 and showed good resolution.
Troubleshooting: Uneven quadrants in WAXS Greateyes CCD camera
For at least the past few months, the upper right-hand quadrant on the WAXS camera has not been uniform with the other quadrants. This poses issues in the quantitative analysis of scattering data especially for samples that contain significant molecular orientation anisotropy. It was expected that any heterogeneities across quadrants should be corrected during a dark subtraction, but that has not been the case.
The dark images are being investigated further. It has been noted that when performing energy scans (nbs_energy_scan) with repeat exposures at each energy would result in dark images not being taken at sufficient frequency. The camera temperature drifts across multiple exposures, and having infrequent dark images results in inaccurate dark subtractions, as a dark image may not be representative of the state of the camera several minutes later. However, this was likely not the case for the repeat scans performed during this beam time, which were run at a single energy from nbs_count scans with dark images taken every ~2-3 minutes. In the plot below, the mean dark intensities appear to have low variation. Furthermore, every dark-subtracted image across this range of time showed the same heterogeneity; if unrepresentative dark images were the root cause, then dark-subtracted images closer in time to the dark image should appear more homogeneous than those further in time from the dark image. Further investigation is necessary.
February 13, 2025
The same beam alignment as the February 10, 2025 beam time was maintained, as there was no switch to the NEXAFS station since then.
Sample position and energy calibration
The same sample bar that was used on January 30, 2025 (fiducial positions noted under January 23, 2025) was used for the current beam time, and the correct_bar positions from then were used as the initial guesses. The corresponding scan IDs are 92210-92219.
RE(fly_find_fiducials(f2=[-2.7923677459465375, 1.0046238327463828, 1.0805282896897526, -2.7630402458556915], y2=3.2577556976279864, f1=[-2.2560237681427995, -0.01548851211297432, 0.6519429066640681, -1.544682725471244],y1=-187.36415907091057))
correct_bar([3.1871194905539335, -2.6980924010428287, 1.0956161605448378, 0.9907430754972397, -2.687595808548622, -187.4587961158209, -2.185916069124269, -0.09688456871974382, 0.6279687074497302, -1.4404897259656122], include_back=False)
February 10, 2025
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.2 | 0.2 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.6 | -0.6 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | -0.55 | -0.55 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.46 | 0.45 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 24.89 | 22.1 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.79 | 7.93 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
Sample position and energy calibration
The same sample bar from November 14, 2024 was used, and the correct_bar values from that time were used as the initial guesses after adjusting the y values due to recent encoder troubleshooting. The corresponding scan IDs are 91693-91702. The automated scans all had good peaks, and those maxima were accepted.
RE(fly_find_fiducials(f2=[-6.931659852522429, 1.116506322622492, 5.347, -3.1815447384910085], y2= 2.28566435717, f1=[-2.327198181209185, -0.2716089944326285, 0.4044408887018089, -1.968],y1=-188.072338831))
correct_bar([3.001401049919686, -6.624193837973033, 1.1136431590316664, 4.877709483702284, -2.87939794167864, -187.2336437866326, -2.145700793431706, -0.005210955306082522, 0.5161157444447216, -1.5915288734209383], include_back=False)
Energy calibration scan IDs are 91703-91717.
January 30, 2025
Energy calibration
Several HOPG scans were run during this beam time. When comparing an HOPG scan from this beam time (e.g., scan ID 91657) to an HOPG scan from 2024 with the same slits 1 vsize (0.02), sample rotation (70° from normal incidence), and beam polarization (90°), such as scan ID 84990, the former shows weaker energy resolution. It is unclear if this is due to the use of the I400 electrometer or becaue of a compromised energy calibration.
Beam stability
Open-beam scans were performed at various times throughout the beam time. At the fluorine K-edge, some features are seen at ~690 eV on both the I0 and photodiode signals, suggesting that there may be fluorinated contamination on the upstream optics.
At all edges, and especially at the S L-edge, the I0 signal appears to be noisier than the photodiode signal.
It was expected, based on open-beam scan data from prior beam times (e.g., May 14, 2023), that the fluorine K-edge would have higher flux than the oxygen-edge. However, that is not the case in the data below. Furthermore, the data from May 14, 2023 and November 2023 suggests that the beam flux should be increasing across the fluorine K-edge, but it is decreasing in the plots below. These discrepancies could suggest that the energy calibration is not optimal such that the M2 and PGM angles are not properly synchronized with the EPU gap.
At all edges, the normalized signals did not all collapse onto the same master curve, but there seems to be a cluster of two or 3 master curves. It is unclear if some other beamline conditions changed between these clusters. Still, this is an improvement from pre-September 2024 (e.g., July 14, 2024) when there was no collapse in the double-normalized signals over time.
January 23, 2025
An unusually large number of beam dumps occurred on January 23. The durations of these beam dumps also were longer. Possibly this may be the result of the 500 mA ring current being less stable.
Beam alignment
| Description | Process variable name | Initial values at the start of beam time,
500 mA ring current |
Final values at by the end of beam time,
500 mA ring current |
|---|---|---|---|
| EPU60 front-end slit h size (inboard-outboard direction) | FE:C07A-OP{Slt:34-Ax:X}size | 1.5 | 1.5 |
| EPU60 front-end slit h center | FE:C07A-OP{Slt:34-Ax:X}center | 0.2 | 0.2 |
| EPU60 front-end slit v size (up-down direction) | FE:C07A-OP{Slt:34-Ax:Y}size | 1.5 | 1.5 |
| EPU60 front-end slit v center | FE:C07A-OP{Slt:34-Ax:Y}center | -0.6 | -0.6 |
| Mirror 1 X (inboard-outboard) | XF:07IDA-OP{Mir:M1-Ax:X} | 0 | -0.55 |
| Mirror 1 Y (up-down) | XF:07IDA-OP{Mir:M1-Ax:Y} | -18 | -18 |
| Mirror 1 Z (upstream-downstream) | XF:07IDA-OP{Mir:M1-Ax:Z} | 0 | 0 |
| Mirror 1 Pitch (rotate along up-down axis) | XF:07IDA-OP{Mir:M1-Ax:P} | 0.68 | 0.45 |
| Mirror 1 Roll (rotate along upstream-downstream axis) | XF:07IDA-OP{Mir:M1-Ax:R} | 0 | 0 |
| Mirror 1 Yaw (rotate along inboard-outboard axis) | XF:07IDA-OP{Mir:M1-Ax:Yaw} | 0 | 0 |
| Mirror 3 X | XF:07ID1-OP{Mir:M3ABC-Ax:X} | 26.649 | 22.35 |
| Mirror 3 Y | XF:07ID1-OP{Mir:M3ABC-Ax:Y} | 18 | 18 |
| Mirror 3 Z | XF:07ID1-OP{Mir:M3ABC-Ax:Z} | 0 | 0 |
| Mirror 3 Pitch | XF:07ID1-OP{Mir:M3ABC-Ax:P} | 7.82 | 7.89 |
| Mirror 3 Roll | XF:07ID1-OP{Mir:M3ABC-Ax:R} | 0 | 0 |
| Mirror 3 Yaw | XF:07ID1-OP{Mir:M3ABC-Ax:Yaw} | 0 | 0 |
An image of the beam on the FS1 mask was captured with the EPU60 front-end slits fully open.
Mirror 1 and EPU60 front-end slit parameters were set based on characterization on the FS6 screen. A comparison is shown below between the initial (bottom) and final (top) Mirror 1 settings.
Mirror 3 settings were adjusted accordingly to direct the beam toward center of the RSoXS slit 1 apertures used for normal operation. Although the horizontal position of the beam was centered on the slit 1 apertures, the vertical center of the beam is significantly lower than that of the slit apertures. There is not an easy way to steer the beam vertically, and the slit 1 apertures were not lowered because otherwise, the beam would be off-center on the WAXS camera.
WAXS camera motor issues
Initially, the WAXS camera motor was not responsive and did not move to the set locations. Software enabling/disabling did not resolve the issue. There was no visible amp fault or other errors, but the MC21 motor controller showed a DISABLED/RESET light on the same motor axis. After power cycling this motor controller while troubleshooting other separate issues, the DISABLED/RESET light remained on, but the motor became responsive in software. This motor was not used extensively during this beam time to fully test that it works, though.
Solid sample up-down motor encoder loss --> Resolved
While attempting to move the solid sample up-down motor (PV name: XF:07ID2-ES1{Stg-Ax:Y}Mtr) in increments of 1 mm, a following error was encountered along with encoder loss. In all cases, the motor would move in an intermittent and jerky manner; example videos were taken at velocity = 1 mm/s, acceleration time = 0.5 s as well as velocity = 0.1 mm/s and acceleration = 2 s. Possibly due to the jerking of the entire manipulation mechanism, the inboard-outboard motor also was showing motion when only the up-down motor was moved. While moving this translation manually using a hand crank, the downward motion had significantly lower resistance than the upward motion; likely this is the result of the vacuum force acting on the bellows.
Initially, it was suspected that this was the same issue that had been encountered a few times earlier where the motor shaft would shear off due to a hole being drilled into it. However, this was not the case, as the shaft of the original motor was intact, and installing a brand new motor still caused the same issue. The teeth in the rubber gear also looked intact. The original lubrication looked intact, and further lubrication also did not resolve the issue. The motor controller also was power cycled a few times during this troubleshooting progress.
During all of the troubleshooting attempts, the encoder never updated its readout value, and the readhead showed a blinking red light (a steady blue or green light would indicate no issues) throughout the full translation range when moved using the manual hand crank. The likely cause of this encoder loss was that the readhead mounting bracket was secured too tightly, which caused the bracket and the upstream side of the readhead to abrade against the stage. This abrasion peeled off copper chips from the stage, and some of the copper chips/dust may have contaminated the readhead or the readstrip (which is marked with measurement increments), which would result in the readhead not being able to read the position. This abrasion may be the cause of the temporary following error on October 27, 2024.
To resolve this issue, the readhead was re-mounted to the stage using washers to increase the spacing between the stage and the readhead as well as its mounting bracket; non-metallic shims (spacer strips) were used to ensure that the distance between the readhead and stage was 0.6 mm. The mounting screws were adjusted to attain the desired Yaw, Roll, and Pitch angles of the readhead. The readhead and readstrip were wiped clean using isopropanol and a lint-free cloth, and the readstrip was inspected to ensure it was properly adhered to the stage. To access the tight space on the inboard side of the beamline, a manlift was used from the downstream side, the downstream diagonal struts were removed, and the up-down motor was cranked downward so that the screws on the mounting bracket were not blocked by the upstream diagonal struts. These steps resulted in the readhead showing green and capable of being moved via software. One minor issue is that it takes a few seconds longer for the motor to transition from "Moving" to "Positioned"/"Disabled"; this might indicate that the readhead still needs fine adjustments in its spacing or rotation. This issue also may be due to the longer acceleration time.
The velocity was restored to 3 mm/s and acceleration was restored to 0.3 s, which were the values prior to October 27, 2024. Because of the encoder loss, the absolute position was unknown. The motor was moved to its top physical limit, and this position was defined as 360.042 mm (user offset = 174.5 mm).
For future troubleshooting, the motor scan be scanned in open-loop mode (i.e., the motor is de-energized and does not hold its position through the encoder feedback) through software; the hand-cranking method is the manual equivalent of this. The open-loop mode can be accessed by using the PEWin32Pro2 software which bypasses the EPICS IOC. This software is downloaded on a laptop at the beamline, and this laptop can be connected directly to the MCS8+ motor controller via ethernet or USB. In preparation for open-loop mode, it is good to gather any specifications on the assembly, identify if there is a gearbox, find the full step resolution, and find the lead screw pitch. These motors also have a rotary relative encoder that is normally not connected to the motor; this relative encoding would need further setup. In case the above procedures did not resolve the issue, it is also good to check that the readhead itself can function properly (on a different stage or using a spacer). Finally, it is good to check that the readhead cable cannot snag on anything as the stage moves up and down.
Energy scan testing and monochromator motor issues --> Resolved
The RSoXS codebase is undergoing significant refactoring in preparation for data security upgrades as well as an effort to standardize the codebases throughout the SST beamlines. While attempting to test upgraded energy scan functions, the EPU gap motion was very slow. It took ~1 s for the EPU gap to move 40 um. During these scans, the monochromator reached the desired position in 0.2 s, but then took and additional 0.4 s to report done.
During the energy scans no errors were seen in the monochromator motors. However, while trying to return the energy to 270 eV and polarization to 0° at the end of this beam time, the MirrorPitch, MirrorTranslation, and GratingTranslation motor drives kept getting disabled. Also, the GratingPitch motor drive kept encountering a following error. These errors could be cleared by hitting the Enable buttons, but subsequent movements resulted in the same errors. Further troubleshooting suggested that these errors occurred more often at certain energy ranges (e.g., over 500 eV) possibly because the motors require different speeds at different grating angles. These issues were resolved temporarily by reducing the motor speeds, and then the motors were re-greased to enable motion at the desired speeds. Other troubleshooting attempts confirmed that software settings, voltage for the I/O logic, motor controller, and encoders are working properly.
After a few days, however, a separate issue arose in which the beam energy cannot be changed using Bluesky or Phoebus even though the EPU and M2/PGM motors appear to work well. The issue started after February 2, 2025 at 2:43 am when sulfur L edge energy scans were being performed at ~165 eV; this energy range was scanned several times within the past day without issues. While attempting to change the energy in Bluesky, the movement would show as completed half of the times even though no motors actually got moved, and the other half of the times the command would hang indefinitely. While attempting to change the energy setpoint in Phoebus or PyDM, any number entered (e.g., 180 eV) would get reverted automatically to 165 eV. During this time, the EPU and M2/PGM motors never showed any alarms/faults in Phoebus. Also, the individual M2 and grating pitches could be moved/tweaked using Phoebus, and the energy readback value got updated automatically in Bluesky. By adjusting these individual pitches, the energy was moved to ~428 eV, but subsequently, the same energy setpoint issues occurred in Bluesky and Phoebus; this suggests that the low 165 eV energy may not be the cause of this issue. This issue was resolved by restarting the MC07 motor controller softioc; a similar solution has worked at SST2. In case restarting the softioc were not to have worked, the physical MC07 motor controller could be power cycled, and some variables have to be reset using the PeWIN software, as the power cycling would make the motor lose its coordinate system. Although it has not been confirmed, there have been network scanner issues around similar times in the morning, and this energy issue may be related.
RSoXS main chamber top turbo pump controller stopped working --> Resolved
Possibly due to the power outage that occurred on December 26, 2024, the controller for the RSoXS main chamber top turbo pump shut off and could not be turned back on again. It is being examined by BNL technicians. The following are the specs for the original controller: Pfeiffer Vacuum D-35614 Asslar, Model: DCU 400, Model number: PM C01 823 A, Serial Number: 80033636, Input: [100-240 V +/- 10% 5,2-1,78A 50/60 Hz 450 VA], Output: [48 V - 8.3 A 400 W], Mass: 1.75 kg, Made in Germany, 44406.
It was attempted to install a new pump controller, but the new controller was not able to access the desired settings. The following are the specs for the new controller: Pfeiffer Vacuum D-35614 Asslar Berliner Strasse 43, Model: OmniControl 400, Product number: PE D70 100 0, Serial number: 80100525, Input: [100-240 V +/- 10%, 50/60 Hz, 5.8A], Output: [48 V 12 A, 24 V 3A], Made in Germany, 2021/08.
It is unclear at the moment if the turbo pump itself is intact. For the shorter term, the turbo pump has been isolated to improve vacuum. Prior to isolation, the main chamber pressure was ~6.5e-8 Torr, and the pressure decreased to ~1.5e-8 Torr after isolating the top turbo pump.
The issue was resolved February 20, 2025. The first issue was that one of the cables (a thick gray cable) got disconnected from the pump itself. After the cable was reconnected, the pump was able to run using the new controller. It also was attempted to install the old controller. This controller would light up and allow menu navigation, but the pump could not be turned on; likely something does not work in the controller itself.
Sample position and energy calibration
A new sample bar manufactured in the same batch as the bar used on November 8, 2024 was used. Due to encoder reset in the sample up-down motor, the y positions of the fiducials were shifted. The corresponding scan IDs are 91369-91378.
RE(fly_find_fiducials(f2=[-3.065, 0.3352512723626649, 0.9711543686868644, -2.490692129085892], y2=3.038, f1=[-2.423285319841728, -0.3535216575626793, 0.4478167192604961, -1.8515940405587246],y1=-187.34))
correct_bar([3.2577556976279864, -2.7923677459465375, 1.0046238327463828, 1.0805282896897526, -2.7630402458556915, -187.36415907091057, -2.2560237681427995, -0.01548851211297432, 0.6519429066640681, -1.544682725471244], include_back=False)
Energy calibration scan IDs are 91379-91393.