A bug in the water sampler's electronics has come and gone throughout the iterations of our design: the valves would reset intermittently when the pump motor was operating. First we moved the pump from the same shift register as the valves to its own dedicated MOSFET. We added a small capacitor for mild noise. The problem disappeared until later iterations. Then, we changed the MOSFET to a dedicated H-Bridge motor controller from Adafruit, equipped with its own 22uF and .1uF decoupling capacitors. The problem disappeared until we used a larger pump in a later iteration. So we talked to Jim Wagner about it and he cleared things up for us.
It turns out that brushed DC motors create a lot of noise, which is exactly as we expected but did not know how to fix. The brushes in the motors cause irregular noise in the power lines with pulse widths too small to pick up on a multimeter, but quite easy to see using an oscilloscope. I hooked up the arduino's 5V pin and ground to a probe and ran the system under the conditions that would cause the valves to reset. The scope was set to trigger on a falling edge lower than 3.04v. Sure enough, the initially-open valve closed and water forced its way through the flush valve after building pressure.
Look a little closer at the oscilloscope:
The 5V rail dropped to nearly 0V for about a microsecond, more than enough time to drop the SCLR line to fall to a low state and reset the shift registers. This explains why the valves were resetting! It wasn't just this one instance, either; the valve continuously reset as the command "V1 1" to open it was sent, and the oscilloscope was continuously triggering due to a falling edge like the one in the picture as long as the pump was running.
OK. We had already added a 100 uF capacitor between the Arduino's VIN (12V) and GND pins. I removed that and bumped it up to 1000uF, reconnected the system to the oscilloscope and ran the test again, this time with similar but notably different results.
The valves were still resetting, and the oscilloscope was triggering often, but the voltage never dropped below 2V. A small victory, surely, however the problem remained. I decided to add a 100uF capacitor between the 5V and GND pins in addition to the capacitor that I had already attached between the 12V and GND. A retest resulted in the valves resetting after 8 seconds of stability, but the magnitude of the noise was still the same:
I read here after some quick research that multiple capacitors of different sizes in parallel would buffer different frequencies of noise. So I unplugged everything again and took the board to the soldering station, tacking on a 10nF ceramic capacitor parallel to the 100uF cap between the 5V and GND pins. This time the valves lasted 15 seconds before resetting on a much smoother blip, magnified to 250ns per division below:
The same signal magnified to 10ns per divison below:
So close! I carefully unplugged everything one more time, soldered on a 1nF capacitor in parallel, then set up the test one final time:
Finally, the valves stayed on! The system ran for a minute straight without so much as a whisper from the probed voltage. This solution can be used in the next iteration of the PCB after more research, calculation, and perhaps discussion with Jim in order to protect against a wider range of noise on more lines more efficiently.