Android 86.1.1 client will not connect to custom sync server over insecure connection (http://)

I have a custom sync server running and working just fine with Firefox v86 Windows 10 desktop clients and Firefox v68 Android clients.
I do the following on Android:

Install Firefox
Enable debugging mode
Specify a custom sync server
Sign in via Firefox account
Press Sync Now

I have a tcpdump running on the sync server machine and I can see all the traffic from the desktop clients. Yet, when I press "sync Now" on Android I do not even get a single packet hitting the sync server. Its like the Android client does not even try to initiate a connection at all, and hence I cannot sync from Android.

It does not matter if the server even exists, is running etc. The client will not even send one network packet to the sync server hosts. Seems like a 100% sync client issue.

I have tried all the following URLs, all with the same result ie zero network packets

If possible, please try using https:// URL, I believe there are some checks in the client code that refuse by default to connect to insecure servers.

Sorry, it did also try that. I have tried it again also - but the same result - zero network packets towards the syncserver
So adding the following attempt:

https://server/token/1.0/sync/1.5

So just on this, if there are such checks (allow https only) then that should be able to be worked around to support development environments etc. It is sometimes a valid use case to not use https and the risk can be acceptable.

Same here. I am using HTTPS with a self‐signed certificate (installed on phone). Latest version makes no effort to connect, and instantly stops when trying to initiate manually.

Please try:
https://server/token/1.0/sync/1.5/
That slash at the end is how it worked for me. It was driving me crazy too. I believe I posted this earlier in an issue.
Like this, with proper https (I use caddy-docker-proxy + a cheap own domain) this works on Android 11, Pixel 4a and a Samsung S20.

I added / to the end of the URL but still zero network packets seen from a tcpdump running on the sync server. I have now tried the following URLs with the same result of zero network packets:

Are you sure you added the URL to the sync area, not the Account part? One of the other mistakes I made and didn't notice for days..
This is how it works for me:

Thanks for the screenshot. Yes, this matches what I have tried as per my list of URLs

(Sorry lack of replies here, I don't have a lot of time to look into self-hosting stuff these days).

I think we'll probably need debug output from the Android device to figure out what's going on here. Any chance you can try connecting adb to the phone and obtaining client-side logs? There are some instructions for how to do this on Fenix here:

https://github.com/mozilla-mobile/fenix/wiki/Logging-Crash-Information#using-adb-logcat-to-get-crash-information

(They say they're for getting debug info about crashes, but if you mentally replace "Reproduce the crash" with "Sync now" it should help produce debug logs for the sync process).

I assume that "submit the crash report" is translated to collect the stdout from the adb output.

That output is available here:
https://www.edcint.co.nz/tmp/expire_14/info.txt (please let me know when downloaded as this will expire in 14 days)

What I did:
Start adb command line at Tue 16 Mar 2021 14:39:46 AEDT
Started Firefox
Went to Settings->Account
Pressed "Sync now" several times
End adb command line at Tue 16 Mar 2021 14:40:07 AEDT

(please let me know when downloaded as this will expire in 14 days)

Thanks, I've grabbed a copy to investigate further 👍

@grigoryk

Unfortunately the logs aren't giving me any additional clues, but some notes from my first look at them:

We see several instances of Successfully executed: getTokenServerEndpointURL, which correspond to FxA setting up the sync auth info here. Notably, we do not see any of the warnings that we'd expect to see if this step failed, so I infer that it's working correctly.

But then...nothing. No logs from the sync system at all, except for a single line from syncmanager telling us it's starting up:

I/BgSyncManager(16650): Periodic syncing enabled: PeriodicSyncConfig(periodMinutes=240, initialDelayMinutes=5)

The next log line I would expect to see is here when the syncmanager kicks off a sync. I don't see any of those, and I don't know why - it's logged at debug level but we see plenty of other debug-level log entries in the file, so that doesn't seem like it should be a problem.

So...I can't quite explain what's happening here :-(

I'll see if I can get @grigoryk to take a look at the logs for additional insight.

Successfully executed: getTokenServerEndpointURL

I think this might be telling lies or it is telling only a partial truth. Whilst the timing of those log entries looks about right, a packet capture running on the sync server sees zero packets on port 5000 from the test phone. The same packet capture sees network packets on port 5000 from other firefox clients (Windows, Android Firefox v68). I can't see any reason why I wouldn't see the packets on my network

At a first glance, logs are consistent with us not entering this block: https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L302-L305

... which maybe could happen if the lock we're using for synchronized is never released? this in that code is actually pointing to an instance of CoroutineScope, not the account manager... I'm not sure what that'll cause (who else can hold a lock on that scope object..? for how long?).

Furthermore, it seems like we're never actually hitting syncManager.start call to initialize the dispatcher. See https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L904-L910

A null dispatcher could also be consistent with these logs. (it's initialized as part of that start method - https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/sync/SyncManager.kt#L152-L154)

For this to not be hit, either we're not notifying account state observers when we should as we're going through the state machine, or that AccountsToSyncIntegration instance isn't registered as an observer.

In the logs, it does seems like we're taking the correct path through the state machine, and should be calling the observers at
https://github.com/mozilla-mobile/android-components/blob/4982e9ba1b1a72a149f4137ebaa828ba19ebf214/components/service/firefox-accounts/src/main/java/mozilla/components/service/fxa/manager/FxaAccountManager.kt#L535 - but that doesn't seem to have an effect. getProfile is mentioned in the logs at just the right moment, as expected.

I'll keep looking tomorrow.

@majurgens

@majurgens I've tried out the custom sync server setup locally, using our production instance as the custom server (it worked fine!). Would it be possible for you to try that, and see if that works? This is the url: https://token.services.mozilla.com (as discoverable via https://accounts.firefox.com/.well-known/fxa-client-configuration)

If that works, it's likely that the problem is connected to your local setup. Otherwise there's something off about our custom server handling logic (I just don't see what, yet).

Also, the logcat you've supplied before lacked most of the Debug level logging. I'm assuming you didn't filter out these log statements yourself, right?

Ok, so now it gets interesting.
I set the URL as you suggested (good idea) and can confirm that Firefox client syncs (well, it says it does and I can see network packets). Certainly different to my specified problem.

So now I am looking at what is different between all the URLs I have tried vs that one.

I checked local name resolution and network access by pasting http://server/token/1.0/sysnc/1.5 into the firefox search/URL bar as though this was a website I wanted to visit. I got a JSON response effectively telling me that the request was unauthorized.

So if Firefox can resolve the local name and access it then the other differences I see between the URLs are:

specification of a port number
use of a URI rather than just a hostname

I tried a few more things:

used http://server as the custom sync server URL
Result: zero packets
used https://server as the custom sync server URL
Result: some packets hit the sync server when sync is first configured. Will not send any more packets when trying to manually sync, even after reboot.

So it does appear that it fails under the following conditions:

the use of a URI
the use of HTTP (instead of HTTPS)

At this point I will just restate that, if there are check that allow https only then that should be able to be worked around to support development environments etc. It is sometimes a valid use case to not use https and the risk can be acceptable. I'd suggest that this limitation should be removed.

You mentioned using a self-signed certificate, so I wonder if the initial packets are from trying to connect at the TLS level, but then we don't see any more packets because the certificate is rejected. If you browse to the https:// version of your server (e.g. https://server/token) in the urlbar, does it succeed?

The person trying to use https with custom certificates with me. I actually checked and it seems my previous attempt to access the address only worked because of an exception, not because it was properly detected in Firefox, so this could very well be a relevant problem. I know this is a bit of a tangent, but what is the best way to ensure Firefox on Android correctly reads custom certificates (ergo, what is the procedure to add them to its store)? I've added the certificates in question to the Android store, and have followed instructions for older versions of Firefox for Android that involve putting the .crt files on a website and serving them with MIME application/x-x509-ca-cert, but that no longer seems to work (this just downloads the crt files). Could you point me in the right direction? If so, that may solve my syncing issue, though I'd have to test again to be sure.

Why don't you simply use Sync as it is attended and obtain proper https certificate from an authority like letsencrypt for your domain, your reverse proxy can take care of that easily. If you use Docker, the setup can be extremely simple with just 2 containers, one for Caddy and one for Firefox-Sync. Nothing else needed except for your own properly redirected domain:
https://github.com/zilexa/Homeserver/blob/5bbb45d282aa2ffd8f97023d999dcaa9224bc2b6/docker/docker-compose.yml#L88

Notice how the Firefox container only needs 3 labels and that's really it.. zero other config is needed, that includes caddy, what you see there is everything.

The time you spend on documenting this issue seems like a waste when a properly configured server works just fine with Firefox Sync..

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

And after configuring server to ssl (own ca, ca certificate installed on android) neither old nor new versions of Firefox sync. Old version shows no connection in server log, new Firefox reports 401 error (is at least able to connect server).

Firefox 68.11.0 - worked with http, certificate seems to work in browser (I do not remember, if I needed to accept it in Firefox), sync shows not connections in apache log (security.enterprise.roots.enabled = true).
Firefox 87.0.0-beta.6 - can open web page, attempt to sync results in 401 error in apache log on GET https://SERVER/PATH/sync/token/1.0/sync/1.5 (security.enterprise.roots.enabled = true).

Firefox 86.1.1 - ignores android installed ca certificate, even after manually opening page on server and allowing certificate refuses to sync. After every restart needs to accept certificate opening page again. No about:config, cannot check security.enterprise.roots.enabled value.

Desktop Firefox works with https, ca certificate was installed to firefox.

what is the best way to ensure Firefox on Android correctly reads custom certificates

I'll ask around and see what I can find out.

No about:config, cannot check security.enterprise.roots.enabled value.

From folks on the Fenix team: you do still need to check security.enterprise.roots.enabled in order to use the Android cert store, but in order to do that you need to be running Firefox Beta or Firefox Nightly, because about:config is disabled in Firefox Release.

There is also a tracking bug for letting you enable this on release here: https://bugzilla.mozilla.org/show_bug.cgi?id=1678191

I suspect, that current issue with newest beta versions is #263. Symptoms are the same. Only thing remaining is: why Firefox 68.11.0 is unable to connect to https sync server...

After creating new install of sync server Firefox 87.0.0-beta.6 works (with enterprise roots enabled) with ssl. issue was probably old syncserver version (last time I tried update, it was complete failure and sync server became unworkable (I needed to restore from backup to working state).
Latest stable needs security.enterprise.roots.enabled support.
And Firefox 68.11.0 still does not show up in apache logs, when attempting to sync. I'll try to upgrade to latest stable and hope, that security.enterprise.roots.enabled survives the transefer.

And this setting survived the upgrade. Following that bugzilla bug about that setting.

Just a note to tell you what I have ended up doing.

I installed a Letsencrypt SSL certificate and ran the sync server behind an apache httpd frontend. I am now using the following format URL successfully:
https://server/token/1.0/sync/1.5

I did also try https://server but that did not work.

Note that https://server/token/1.0/sync/1.5 did not work at first as my first attempt (can't remember the exact details). I had the client setup with https://server/token/1.0/sync/1.5 but the server was not working. It would not work again until I signed out, changed the URL to the same thing and signed in again. I am guessing but maybe the client registered the server as dead and some part of the signout/URL change/signin made it reattempt.

A little more detail on what I did that might help others with internal servers.

I am on Fedora 30 (old version so that the python 2 sync server works better)
I decided to use Apache HTTPD frontend
I have a public domain mydomain.com hosted on a Cpanel service
I made an internal DNS entry server.mydomain.com
dnf install certbot python-requests certbot-apache
cd /opt
git clone https://github.com/letsdebug/certbot-cpanel-dns-auth-hook
-- this is a Cpanel client that can dynamically make DNS entries on the Cpanel hosted server. You need to be able to do this for internal servers since the only way you can register a Letsencrypt certificate is by using a DNS check. The HTTP check will not work because Letsencrypt serves cannot see the internal server
-- if you are using some other hosting you will need to find a way to make dynamic DNS entries at your DNS hoster
Now get the certificate
certbot certonly --manual --manual-auth-hook "/opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py create" --manual-cleanup-hook "/opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py delete" -d "server.mydomain.com" --preferred-challenges dns-01
during this command the script /opt/certbot-cpanel-dns-auth-hook/cpanel-dns.py adds a DNS TXT entry on my public DNS for _acme-challenge.server.mydomain.com, loads it with a key, Letsencrypt servers check the key (to verify you own the domain), then the scripts deletes the DNS entry
this places the certificate into my file system
now all I have to do is make sure my Apache config points to that certificate
I used the follow Apache config (added this to ssl.conf)

<Directory /opt/mozilla-sync/syncserver>
  Require all granted
</Directory>

<VirtualHost *:443>
  ServerName server.mydomain.com
  DocumentRoot /opt/mozilla-sync/syncserver
  SSLCertificateFile /etc/letsencrypt/live/server.mydomain.com/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/server.mydomain.com/privkey.pem
  WSGIProcessGroup sync
  WSGIDaemonProcess sync user=apache group=apache processes=2 threads=5 python-path=/opt/mozilla-sync/syncserver/local/lib/python2.7/site-packages/
  WSGIPassAuthorization On
  WSGIScriptAlias / /opt/mozilla-sync/syncserver/syncserver.wsgi
</VirtualHost>

Configure sync clients to use https://server.mydomain.com/token/1.0/sync/1.5 (which works internally only because if your internal DNS entry)

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

This is not true. As long as the internal server can get out to the internet, it does not matter if it cannot be seen from the Internet. The Letsencrypt HTTP will not work (requires server that is visible from the Internet) but the DNS will work if the server can get to the Internet. My previous comment in this thread goes into some detail on how I made this work

Letsencrypt is no use, if sync server is internal network only accessible from Internet via VPN.

This is not true. As long as the internal server can get out to the internet, it does not matter if it cannot be seen from the Internet. The Letsencrypt HTTP will not work (requires server that is visible from the Internet) but the DNS will work if the server can get to the Internet.

Only, if service provider has DNS update API and that API is supported. At least that is my understanding.

It is my understanding, that Letsencrypt DNS requires, that DNS provider has API to update DNS data. Anyway, browser should support custom CA-s anyway (Desktop Firefox supports it by installing them directly in Firefox). Anyway, that is Firefox issue and not syncserver one.

Navigation Menu

Repository navigation

Android 86.1.1 client will not connect to custom sync server over insecure connection (http://) #269

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Android 86.1.1 client will not connect to custom sync server over insecure connection (http://) #269

Description

Activity

rfk commented on Feb 26, 2021

majurgens commented on Feb 26, 2021

Atera commented on Mar 13, 2021

zilexa commented on Mar 15, 2021

majurgens commented on Mar 15, 2021

zilexa commented on Mar 15, 2021

majurgens commented on Mar 15, 2021

rfk commented on Mar 16, 2021

majurgens commented on Mar 16, 2021

rfk commented on Mar 16, 2021

rfk commented on Mar 16, 2021

majurgens commented on Mar 16, 2021

grigoryk commented on Mar 16, 2021

grigoryk commented on Mar 16, 2021

majurgens commented on Mar 16, 2021

14 remaining items

Atera commented on Mar 17, 2021

zilexa commented on Mar 17, 2021

virgoparna commented on Mar 17, 2021

virgoparna commented on Mar 17, 2021

virgoparna commented on Mar 17, 2021

rfk commented on Mar 17, 2021

rfk commented on Mar 17, 2021

rfk commented on Mar 17, 2021

virgoparna commented on Mar 18, 2021

virgoparna commented on Mar 19, 2021

virgoparna commented on Mar 19, 2021

majurgens commented on Mar 23, 2021

majurgens commented on Mar 23, 2021

virgoparna commented on Mar 24, 2021

virgoparna commented on Mar 30, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions