Even better is to have three lines on the charger. One to power the circuit, one to charge battery pack number 1, and one to charge battery pack number 2. By isolating the battery packs from each other and from the main circuit charging can be done more rapidly and with greater safety.
The common technique for isolation is by diode ORing the batteries. However, this is pretty power hungry.
There exist some very nice devices used to create an ideal (30 mOhm) diode for this purpose (e.g., Linear Technologies, LTC4358 - http://cds.linear.com/docs/Datasheet/4358fa.pdf). These devices are invariably surface mount and would be far easier to use if you would create a breakout board for them.
Thanks for taking a look.