The Edge of Emulation
The Edge of Emulation2020-01-20 18:01:06
SNES emulation has come a long way from its inception in the 1990s.
These days, I am able to keep my bug tracker free of game-related issues approximately 95% of the time, and new issues are usually very minor and quickly resolved.
We are in a good place with our emulation of nearly all SNES components, with the sole exception of the SNES PPUs, or video processors. When it comes to scanline timing, we have more than perfected that, but when it comes to the individual pixel timings, needed for raster effect accuracy, it is the one weak area left in SNES emulation.
In practice, I can only name one game (Air Strike Patrol) that intentionally relied on SNES raster (mid-scanline) effects, so the importance of this is easy to downplay, however for the sake of completion it represents the final frontier of SNES emulation research.
When I first entered the SNES emulation scene in 2004, things were in bad shape: aside from high-level emulation of SNES coprocessors, no substantial core progress had been made since around 1999 or so.
Emulators back then worked on a principle I like to call whack-a-mole emulation: whenever a game wasn't functioning, a fix would be devised that would seemingly make that specific game work, but the fixes weren't being verified as accurate to real hardware. Over time, the fixes would turn out to be wrong and break other games. Fixes for those games would break yet more games. And then fixes for those would break the original game that started this cycle. Rinse, repeat.
The thing I knew we needed from day one was to not prioritize fixing specific games, but rather to understand how the actual hardware operated, and to implement correct, hardware-verified fixes.
The key insight that made this possible was me devising library code that could seek to exact cycle positions within each emulated video frame, execute code of my choosing, and then analyze the results. By making loops over ranges of cycle positions, it became possible to analyze and determine exactly where various operations on the SNES occurred.
This worked amazingly well, and combined with the help of others over the next fifteen years, we were able to basically perfect the internal operations of the CPU (central processor) and SMP (audio processor) inside the SNES.
Some time later, a developer by the name of blargg appeared to tackle the SNES DSP (audio generator), by taking advantage of cycle-timed SMP code and a key feature of the DSP: the echo buffer. The DSP would write all generated audio samples into this echo buffer, and from here it was possible to analyze the results of audio mixing, allowing us to once again reverse engineer and perfectly emulate every internal operation of the SNES DSP.
After a successful coprocessor decapping project, I was able to emulate all of the SNES coprocessors using low-level emulation.
And finally to cap things off, several important new hardware test ROMs and a lot of hard work enabled me to emulate cycle-level timings of the SuperFX CPU's instruction and pixel caches, and bus-conflict arbitration stall timings of the SA-1 CPU.
Thousands of additional hours into other coprocessors (decompressors, real-time clocks, memory mappers, etc) has led to a state of near-perfect SNES emulation.
That is, with the one exception of the SNES PPUs.
To generate video, the SNES relies on two PPU (picture processing unit) chips, the PPU1 and PPU2.
Unfortunately, unlike every other processor, these have the critical challenge that they act as black boxes to the SNES itself: you cannot analyze the pixels it generates, as they are sent directly from the PPU2 out through the analog MultiAV connector on the back of the SNES console.
Where the CPUs had registers, flags, I/O ports and memory to analyze results, and the DSP had the echo buffer to analyze audio generation, the PPUs have no such functionality.
The only feedback the PPUs provide are range-tile over flags to inform you if there were too many sprites (or sprite tiles) on a scanline during a video frame. Put simply, it's not enough information.
During active screen rendering, the video RAM of the PPUs is completely inaccessible to the SNES CPU: you cannot even read what is being fetched while the PPUs access this memory.
Either due to an oversight or a lack of concern, the object attribute (sprite) memory and color generator (palette) memory are accessible, but with an interesting twist: the PPUs drive the address bus. By reading and writing to OAM and CGRAM during rendering, we can peek slightly at the internal operations of the SNES PPUs.
We were also able to consider the bandwidth limitations of the SNES PPU design: how many bytes of video RAM could it access at any given time? What would be the smartest pattern to perform these accesses? And so forth.
This led us to a mostly complete general understanding of the SNES PPUs, and with all of the above information combined, I was able to devise (and over the years refine) the first cycle-based SNES PPU renderer.
It's enough to faithfully recreate the raster effects in the one game that uses such effects, as well as to handle nearly all officially licensed software that accidentally modifies PPU registers during scanline rendering, but it's never been perfect: things have always been off by a few cycles, causing raster effects to either start or end a few cycles early or late.
Again, raster effects are so infrequent that you would never notice them. My bsnes emulator even defaults to a scanline renderer for most games because it is so substantially faster and indistinguishable.
But when we're talking about perfection, the truth is, even higan's cycle-based PPU renderer is not perfect.
Recently, a tester named Max has gone through the entire SNES library looking for any and all issues, and indeed, quite a few smaller cycle-based PPU rendering issues emerged. Some were more serious than others and an obvious fix was available. But some were more stubborn.
I should add that from 2004 - 2018, I was mostly alone in pursuing cycle-based timings of the SNES PPUs. Or at least, the only person publicly working on the problem and sharing my notes and source code with everyone.
As of 2019, a bit of a revival has occurred in the SNES emulation scene, and we now have two other open-source SNES emulators under development with a focus on accuracy. I really couldn't be happier about this: I've mostly reached my limits of what I can do on my own, I'm not going to be around forever, and having more people in this space is always a blessing.
Unfortunately, I feel like this has created some pressure at solving these final PPU issues in a quick-and-dirty, unverified approach. In other words, the PPU version of whack-a-mole.
I watched this approach play out fruitlessly for years before I finally stepped in to begin working on SNES emulation back in 2004, and I don't want to see it happen again with this final piece of SNES emulation.
We know that there are 52 writable PPU registers, and 12 readable registers. We also know there is VRAM, OAM, and CGRAM memory.
But there are also an unknown number of internal registers that the PPU uses. These are generally referred to as latches: during screen rendering, the PPU will at various points read these PPU I/O registers, copy (or latch) them internally, transform them in various ways, and then use and update the internal latched values throughout rendering each scanline.
We are completely blind to these internal latches. The way we find out about them is through debugging games with rendering issues. For instance, Air Strike Patrol writes to the BG3 scroll registers mid-scanline to perform a text rotation effect on text in the middle of the screen, while not disturbing a HUD display on the left on the same background layer, so we know these registers can be modified mid-scanline. And then there is an effect in Full Throttle Racing that writes to the BG3 horizontal scroll register around cycles 40 - 80 of the screen rendering to scroll in some text, which unless we latch the fine-X scroll bits (the lower three bits of each tile) prior to the first write, the effect does not render correctly.
Thus, we are able to deduce that the PPU is latching the fine-X scroll registers of the background registers sometime before cycle 40 of each scanline. But we don't know on exactly which cycle this happens. And so we guess.
If we latch too late, after cycle 40, Full Throttle Racing will glitch. But if we guess a value too soon, say on cycle 0, that might be too early, and it might cause issues in another game.
We would have to retest the entire SNES library to be sure such a change did not cause any regressions, which just isn't practical with a library of 3,500+ game titles.
Rather than blindly guess things like this, we need to confirm the exact cycle position on real hardware, and do it right the first time.
See, on the surface, it seems pretty easy to simply tweak values to get the desired results. Even if not perfect, we'll surely get closer each time to the correct answer, right?
But nothing in the SNES PPU is this trivial, and every effect is influenced by other effects.
Imagine the expression:
3x + 7 = 22
It's easy to deduce that x is 5 in this case. We could say that x here was the BG3 horizontal scroll latch cycle.
But now let's say you have the expression:
2x + y = 120
This could be solved if x were 30 and y were 60. But it could also be solved if x were 10 and y were 100. Or if x were 20 and y were 80. And so on.
The SNES PPU's internal state isn't one variable at a time in isolation: it's hundreds of variables all operating in unison.
Imagine a mathematical expression complex enough to generate an entire row of 256 or 512 pixels, across four background layers with differing modes and priority bits, 128 sprites with their own priorities, complex color blending modes, two windows that can be combined with boolean algebra, a color window on top of that, mosaic effects, offset-per-tile effects, and on and on and on.
The combination of possible answers that would render a scanline that looks correct is basically infinite.
At this point in SNES emulation, we've come about as far as we can with guessing what this equation might look like based on the above observations I've mentioned, and thousands of hours of empirical testing, but we will never get this perfect using this methodology: it's a lost cause.
So that's the problem scope, now what can we do about it?
The first, and most arduous, approach is to analyze the analog RGB output: we write very specific SNES test ROMs that try to write to SNES PPU registers at a different cycle on each scanline, and carefully design what should be rendered around this time. We see the visual results onscreen, and by trying to exactly match the output we see on our CRTs (or XRGB-Mini capture devices), we slowly reverse engineer the existence of internal latches and their latch cycle positions.
The major problem with this approach is that it has to be done manually! We have to create a very clever test ROM that relies on generating a visually significant pattern, and once we've emulated the pattern, we can't quickly validate the behavior again.
The best bet would be to save emulator screenshots, and update our emulators to run these test ROMs and compare the video output to cached screenshots in order to perform regression testing.
We will absolutely need regression testing because any future changes are very likely to break previously passing tests.
But even if we do this, we're still relying on the analog video output from the SNES, and we can't perfectly match our screenshots to this output, because the output is analog and thus imperfect.
Not being able to do millions of tests iteratively and automatically makes this process unbelievably slower to the point that I really don't see it being a practical approach.
Many PPU operations (such as full sprite evaluations) are so complex that we could easily need millions of tests to fully understand the behavior, and we just aren't going to be able to visually eye the results of millions of test runs to check for correctness.
I think our best bet is going to be some form of PPU1 and PPU2 pinout monitoring. The two PPUs connect to each other and to the rest of the SNES with address buses, data buses, and more.
If we have the ability to snoop those buses during each cycle of operation, we can see all of the external timings of the PPUs. And with that, we can suss out on which cycles various background tilemaps and tiledata are fetched, when exactly sprite fetches occur, etc.
The easier way to do this is with a logic analyzer, but the problem here is that you still have to write the test ROMs, and run each one. So if you need to evaluate a million combinations of sprite pattern layouts, you're going to be doing a million tests with your logic analyzer.
It gives us better and more reliable information than looking at fuzzy, analog video frames, but it's still not enough.
If we could recruit a talented electrical engineer, I believe that a custom PPU breakout board could be designed that would aid us substantially in reverse engineering.
The idea would be to make a board containing the two SNES PPU chips, and the two external VRAM chips. The internal OAM and CGRAM would come with the PPU2.
We would then construct an interface to connect this board to a PC, and allow us to write software that would program the VRAM and PPU registers to a certain given state, and then run the chips, logging the values on each of the data buses during every cycle of operation, feeding the results back to a computer.
We could then modify our emulators to generate similar logs, and use this as a formal validation to create an emulation that is pinout-compatible with the original SNES PPUs.
This still wouldn't directly tell us about the internal register latches, but it would be so much available information, that we could programmatically generate and test (eg millions of tests as discussed previously), that I think we could mostly figure out the internal operation from this.
The final approach I can think of would be for us to acquire high-resolution die scans of the two SNES PPU chips, and then formally recreate transistor-perfect copies of them in a Verilog / VHDL-like language.
This would reveal to us all of the internal latches, and truly perfect emulation would now be possible: directly on FPGA emulators, and very rapidly in software emulators as well.
We currently have 20x decapped die scans of the PPU chips, courtesy of John McMaster, but we would need at least a 50x die scan (and preferably 100x) in order to trace out its logic.
Call For Assistance
The problem is, aside from the deeply flawed analog RGB method (which I simply don't have the time for), I can't really do this. And I don't see anyone else working in open source emulation that is willing to do this either.
If you can help, please get in contact with me. If we can solve this last problem, we can have almost completely perfect SNES emulation, to a state where even I could be satisfied that the SNES has been preserved for future generations.
If you can't help, I still appreciate you reading this, and hopefully it's now more clear the final problems we have with SNES emulation, and why it's so difficult to solve.
Thank you for reading!
Permalink • 44 Comments
Geoffrey Frogeye2020-01-20 20:56:21Nice write-up. It gets to the point without too much background required nor unnecessary exposition.
I did not understand what was wrong with the logic analyzer method. As long as we can automate them, the number of tests required should not be a problem? Is there something preventing that (maybe: limited inputs pins for the logic analyzer compared to the outputs of the PPUs, inability to load test ROMs automatically...)?
I'm wondering how SNES game developers did to use those specificities in their games. Did they have documentation we don't have access to publicly today? Or did they do some kind of reverse engineering/trial and error to get the effects they wanted?
Mattiac2020-01-20 21:16:31Didn't Dr. Decapitator decap the SNES PPU?
Anonymous2020-01-20 22:59:24This was certainly very interesting to read.
Pardon me if this is a stupid question, but would it be possible to ask someone at Nintendo/Ricoh/whatever how the PPUs work?
Hellmo_Luciferrari2020-01-20 23:50:47Thank you for your hard work and dedication to accuracy.
bry2020-01-21 00:58:49I thought the PPUs were already decapped?
To butcher a phrase... if you have a problem, and you decide to solve it by decapping some chips, now you have two problems.
Brent2020-01-21 04:01:17I have nothing to offer but my appreciation for what you do. This puts so much into perspective for something I often take for granted and really shows how close, yet far away, we really are from unprecedented perfection of emulation. This level of preservation is crucial to this system's place in the future.
jay2020-01-21 05:27:41Is this something (specifically decapping but the other options as well) that someone could be hired to do even if they do not have a particular interest in SNES emulation? e.g. retroarch has a bounty program that people can contribute towards the reward. Eventually (hopefully) the reward becomes large enough for someone to dedicate the time to solving.
Christoph Helms2020-01-21 06:41:23This is a great article and I learned a lot here. How flawed is the current fpga implementation of the SNES? Does it display the same ppu problems as software emulation?
Sean2020-01-21 06:55:28Thank you for all your hard work!!!! As someone growing up with snes and agrees with persevering these games for future generation, I just want to say thanks for everything!
Gaming Blows2020-01-21 08:16:44This is an amazing write-up, and your work is highly appreciated and respected by all of us. Thank you for what you've done and continue to do.
byuu2020-01-21 10:03:59Geoffrey, the logic analyzer method could work. We would just need someone willing to do it that was also up for sharing their results publicly. I don't own a logic analyzer myself, and I'm not sure how to really use one. Air Strike Patrol developers probably guessed that you could change the registers mid-scanline, and tweaked the timings until the results were visually pleasing. The other developers all did so by mistake rather than intentionally.
Mattiac and bry, no but it would seem John McMaster did. Unfortunately they're only 20x magnification, which isn't enough per furrtek to try and reverse-engineer logic blocks out of the images. I'll update the article with this though, thanks!
Anonymous, extremely unlikely they would share.
jay, the specialization is too high, we'd likely be paying hundreds of dollars an hour to hire someone. And it's a task that would take hundreds of hours.
Christoph, the current MiSTer implementation is based off of bsnes' implementation, but there's almost certainly subtle differences between them at this point as we've both diverged a lot from the initial port by srg320. But yes, we're both guessing how all of this works currently. Everyone in SNES emulation is.
Everyone else, thank you for the compliments ^-^
R.Blasta2020-01-21 11:03:05Very informative, I regret I don't poses the required skill set to help,
Still I'd like to thank you for all the efforts made
SNES emulation is how I experienced many games for the first time
When I was in highschool I used to load emulators onto the server
SNES was always a huge hit
Keep up the good work do it right and most wont know you've done anything at all
Doesn't mean we didn't appreciate it :)
Christoph Helms2020-01-21 13:30:27This is a great article and I learned a lot here. How flawed is the current fpga implementation of the SNES? Does it display the same ppu problems as software emulation?
Laurent Giroud2020-01-21 13:38:38Great article, thanks for sharing this information.
> Anonymous: extremely unlikely they would be willing to share
This is just an assumption. And it only costs the time to send a few emails to verify. Surely with such a small cost and such a large payout in case of positive answer it is worth trying, even with (assumed) very low probability of success.
Also I agree with another commenter that logical analysis can be automated, just like analog instrumentation. The Dolphin team has done precisely that to systematically test for regressions.
Gigsworth2020-01-21 16:56:38Are the mcmaster decapped ppu's still available for scanning at a higher power?
tjmurphy2020-01-21 17:19:04Hi there,
I don't have experience with the SNES hardware specifically, however I do have experience with PCB design. My day job is in working as a satellite engineer at a university, where part of my responsiblities is PCB design, procurement, and assembly for prototype applications. I commonly draw up a board in Eagle, order it from China, then order all the parts online and solder it together. Then I do testing, and occasional firmware development.
If you have documentation on the chips in the SNES, a list of precise requirements that the board you describe would have, and would be willing to provide ~$50 toward the cost of the board and components (plus the SNES-specific components), I may be able to hook you up with the Custom Circuitboards thing you described here.
If you want to chat further about this, feel free to email me.
Best of luck in working out what you're looking for, in any case! Maybe you'll find someone with SNES hardware experience, which would definitely be a better pick than me.
Nadia2020-01-21 18:27:11Are the SNES PPUs internally or externally clocked? If they are externally clocked making the breakout board with an onboard logic analyzer and logging facility shouldn't be too hard (like interfacing with the host PC over USB sounds already more annoying than the rest), and I could whip out a quick hardware proof of concept in KiCad in a couple weeks. And ordering custom PCBs from china is easy enough.
If they are internally clocked and the data/address bus clock too fast for the standard microprocessor to keep up I think an FPGA to quickly move samples to a dedicated RAM unit that gets read and sent back by the main microcontroller would work, but my experience there is way weaker.
Dezgeg2020-01-21 18:54:38Are you aware of the project Gekkio did for GameBoy: https://github.com/Gekkio/gb-hardware - maybe he has some insights on how to proceed.
Camel2020-01-21 21:36:23For decapping the chips I'm sure Zeptobars would love to do it if you sent them the chips. Ask them!
micro2020-01-21 22:22:06I have a friend with access to a 48x and potentially another acquaintance with access to something closer to 200x. The friend won’t be able to decap, but if someone else can recap and send chips my way I can get some higher res scans for y’all.
chilliron2020-01-21 22:53:55tjmurphy, and others who may be interested, I would be more than happy to contribute funding for something like this.
Wilson2020-01-22 01:34:42Congratulations for all your contributions, it's a great pleasure to have Nintendo games almost perfectly emulated nowadays. I don't think of me as a cheap pirate. I bought many games in my life and still continue contributing paying as much as I can for all the hard work developers put in their games. But I would never ever had played 1/100000000 of all gaming library if it wasn't by emulators.
When I was a kid, I used to do rom hacking. You know, emulators began to offer tools for us to manipulate memory addresses and coding, translate texts, create IPS and a lot of funny stuff. Today I don't have much time to play (work, marriage and kids), but I would like to thank you and all the emulation community for all the good things. We live wonderful times!
John Smith2020-01-22 03:34:25What would be approximate decapping cost?
kammedo2020-01-22 05:08:57Hei byuu! Awesome! Congratulations for.your whole work!
Gregory2020-01-22 06:10:53Decapping chips is pretty easy (nitric a is & acetone is all you need) so that plus a decent magnifying camera seems like a good way to go, no?
Shivansh Vij2020-01-22 06:53:43I don't have as much experience as tjmurphy, but I also spend a lot of time designing circuits and sending them off to China to be made - though mostly for university robots as opposed to satellites.
I'd also love to take part in the custom PCB if there is documentation on the connection circuitry we'd recreate.
Horde2020-01-22 07:27:16I mean.. your first option seems more fitted to machine learning than human reference. Frame perfect matching with guess and test us the core of ML. It would be significantly faster that way. Perhaps faster than the other two.
fmahe2020-01-22 09:20:44Thank you for this very clear exposé of the problem.
The design files for all these 20 or 30+ years old, abandoned chips is probably sleeping somewhere, collecting dust. I wholeheartedly support the reverse-engineer efforts of the retro-gaming community, but I cannot help feeling a bit sad when I think about the countless hours of work that could be saved with more sensible IP laws. Beyond gaming, chips are in many aspects of our lives now. What does it say about our society and its future if today's bright people have difficulties replicating yesterday's technologies?
notme2020-01-22 11:39:24I wonder if this information isn't available in e.g. the devkit documentation. Surely one should be available from somewhere - a developer with an old devkit left in their possession after the studio closed. You could also try asking the programmers who worked on those games. While the Nintendo or the chip manufacturers wouldn't answer, programmers are usually keen to share information, anonymously if needed.
Jeff Johnson2020-01-22 13:37:19Great article! I wish I had the skills and time to offer assistance.
I want to say thank you to you and all the emulator developers!
Alpha642020-01-22 14:43:00What about the (what I assume is) the entirely software-based SNES classic console? Would it be easier to reverse engineer the software running there compared to the hardware in the original? This assumes that the SNES classic actually uses these components in some meaningful way that would require Nintendo to implement them fully....
Zeek2020-01-22 15:01:31Amazing, really I am stupefied by this.
Sho2020-01-22 15:30:25Maybe a dumb question, but would it help to figure out how the latch works internally by testing a fully working SNES against a specific SNES that has been failing the “EXT Latch” test in the Cart Tester program?
I know I’ve come across quite a number of consoles over the years that failed this specific test and had no clue what it meant.
InvisibleUp2020-01-22 15:54:36Hello again byuu!
If it is helpful for you, I'm an electrical engineering student in my senior year at a large university. I've assisted in firmware development and (very basic) PCB design and testing for a CubeSat for a research team I'm on. I'd love to help with the custom circuitboard project, if you're interested. If you want to chat about this some more, feel free to email me or DM me via Twitter again.
Justin2020-01-22 16:42:56I really dug this write-up, even if a fair bit of it flew over my head. Kudos to you for making it possible to access for folks like me with no knowledge of the inner workings of emulation though. Hope something comes out of the woodwork to help you out. Your work is super appreciated!
Henry Kikenovic2020-01-22 18:51:32Thanks for all the hard work and the attention to detail. I hope someone soon joins you to help complete the last pieces of the puzzle.
Proud AdNauseam Advocate2020-01-23 14:03:11Best of luck, byuu! I'm looking forward to finally seeing the internal workings of everything the PPU has to offer. Especially that mysterious "HVCMODE" pin that both PPU chips have.
Alpha64, the SNES Classic Edition is by far the worst option for analyzing the original SNES's PPU. Its internal emulator, canoe, has game-specific hacks for all 21 games in each region, including Super Mario World of all games. It even hacks the game ROMs to strip out all DSP audio samples and replace them with "PCMF" codes to take a shortcut when emulating the DSP. In short, canoe was written to play games, not to preserve the SNES hardware.
sureanem2020-01-23 15:03:00>But even if we do this, we're still relying on the analog video output from the SNES, and we can't perfectly match our screenshots to this output, because the output is analog and thus imperfect.
>The problem is, aside from the deeply flawed analog RGB method (which I simply don't have the time for), I can't really do this. And I don't see anyone else working in open source emulation that is willing to do this either.
How flawed is it, really?
We are dealing with a pixel buffer which gets more or less deterministically encoded in an analog format. There's a limited amount of pixels and colors. We have a lot of "known good" screenshots for games, and can create test ROMs which write arbitrary data to the screen. Commodity hardware can easily digitize the raw RGB signal (30 MHz IIRC) to PCM for further processing.
Isn't it possible then to do something akin to linear regression to find a way to map the signal to a pixel matrix (e.g. raster image) better than a naïve hardware implementation of the CVBS spec and "good" adapters? Or is sufficient information destroyed in the encoding so as to make reconstruction of the signal impossible?
segfault2020-01-23 19:51:16A few people have chimed in about HW design to do reverse engineering. Is there a discord or other medium for discussing / collaborating on this?
Michael2020-01-24 05:55:12Send me the chips, I can shave the tops off.
Surface grinders are just awesome at removing material. 0001 inches at a time.
mz2020-01-24 18:58:48I think you should get in touch with Andre Weissflog
Cole2020-01-24 21:00:19First off, great write-up byuu. I knew next to nothing about SNES emulation, but everything you said was very clear and understandable.
I have experience with die images -> verilog, and have been developing ways to greatly speed up the process. So far, only one relatively simple chip has been converted this way (the AY-3-8500 "pong-on-a-chip" IC) but I'm actively working on new techniques and other chips. Higher resolution scans would definitely be needed, at least 50x, maybe 100x. Images with the metal layer removed would also be very helpful, potentially allowing automatic markup of the components. With those provided it would be very much possible to have a transistor-accurate simulation, which could be converted without too much difficulty into an FPGA core, and also allow for extremely intimate analysis of the hardware. That's the route I'd recommend, and one I'm able and willing to dedicate a lot of time to.
Anyway, no matter whatever you decide to do, and whoever does it, I'll be following closely.
bally2020-02-11 19:06:01I have nothing but respect for your work. I cannot imagine the amount of time you've spent over 15 years working on these problems. My friends will tease me about the way I complain when I know things aren't "theoretically being done correctly" even if they appear to be working but this is the way. Thank you for all your contributions.
Comments are manually moderated. HTML and Markdown are not parsed.
Captcha: Flipper and Ecco are examples of what type of aquatic mammal?