Update: Federation and community creation are now back online!

Hey all, there’s a hack floating around which spreads via federated comments and steals users’ Lemmy auth tokens. Lemmy.world and other large instances have been hacked, so we’re taking some precautions until this is fixed:

  • We’re logging everyone out so that auth tokens reset
  • We’re closing off federation and community creation until this is patched

FYI, there are no indications that anyone on our instance has been hacked. We did find ten comments with the code injection attack, which we’ve now scrubbed. But it’s very unlikely that this will cause harm at this stage. There are several steps between this and hacking the entire instance. (Also FYI for nontechnical users, the hack affected Lemmy logins and nothing else. Web browsers run all websites in a kind of “jail”)

Sorry for the inconvenience – growing pains. Updates to come as we learn more!

      • zen@monyet.cc
        0·
        1 year ago

        hello, waves poignantly to the everyone still here

        latest that I saw, it seems one of the hacked instances sent out some modlog that has a date too far in the future, and those events federated, so now the modlog of lots of instances are broken.

        anyway, just wondering, would it be feasible to turn federation back on with an explicit allowlist, instead of turning it off altogether?

        • dcx@monyet.ccOPM
          0·
          1 year ago

          I’d prefer not to until we install a patch, since the exploit seems viral in nature (compromise one instance, use that to compromise the next, etc). So trusting one is like trusting all

          We’re testing that in dev so we might refederate later tonight. Or maybe tomorrow

          • zen@monyet.cc
            0·
            1 year ago

            I’m thinking mutual allowlists, but I guess never mind.

            by the way, where did you get the patch from, is it from the github issue?

            • dcx@monyet.ccOPM
              0·
              1 year ago

              Yep! It’s a really obvious one, just escape a bit of user / federation-facing input that wasn’t being escaped. 5-10 lines of code or something.

              • zen@monyet.cc
                2·
                1 year ago

                lemmy 0.18.3 is out, and from what i heard, it has a hardcoded 3-day timeout for federation health status.

                so, if an instance is uncontactable for 3 days it is marked as dormant, and no more federation traffic is sent to it.

                you might want to put it in the sop, in case federation ever needs to be turned off again, to try to bring it back up within 3 days.

                otherwise, other instances may mark monyet.cc as dormant, and the remote communities won’t get updates anymore.

                i think the check is scheduled once a day, so perhaps being marked dormant isn’t that permanent, but 3 days plus 1 of lost federation traffic could cause quite a bit of desynchronization.

              • zen@monyet.cc
                0·
                1 year ago

                okay, it’s probably this and a 0.18.2-rc.1 has just been tagged.

                maybe you can keep the mutual allowlists thing in mind, and perhaps also switching signups to require approval. both of these could be on the sop for the next time this happens (they haven’t fixed the jwt expiry thing, so…).

  • aerir@monyet.cc
    0·
    1 year ago

    No wonder I couldn’t see the posts from here today from my instance. Anyway RC2 is out, which should fix this XSS vulnerability

      • aerir@monyet.cc
        0·
        1 year ago

        Wouldn’t be too fussed about it tbh, can never play it too safe when it comes to such incidents.

        edit-mmm still can’t see the posts here from my instance, sadge

        • zen@monyet.cc
          1·
          1 year ago

          edit-mmm still can’t see the posts here from my instance, sadge

          this is why I wanted allowlist-federation to friendly instances (like yours), instead of turning federation off completely.

          when I saw that it was off completely, I had a gut feeling that it would not be easy to bring back up, as I have not seen any other instance doing it this way (turning federation off completely).

          so, now troubleshooting time, curl test commands from the docs no longer work (they are all returning 404 for me), even webfinger, (which afaik, is an important enabler of federation with mastodon) doesn’t work.

          I hope this can be fixed by rebooting the server, and it’s not something more serious like turning federation off causing all the activitypub private keys to be deleted from the db.

          hate to bother you so early in the morning @[email protected], but I don’t know who else is part of the tech team.

          reminder: please do not copy the nginx config that is above the curl commands on the join-lemmy.org page. they contain a subtle bug. use what is in the lemmy-ansible repo, which has the bug fixed.

          anyway, I don’t think you need to touch anything in the nginx config if you’re following lemmy-ansible completely without any change. if the curl commands return html, then something is wrong with the config, but here they return 404.

          • dcx@monyet.ccOPM
            1·
            1 year ago

            Thanks for the heads-up! We’ve given it a reboot and things seem to be working as intended now. In general I have a weekly crunch period which happens around Tue-Thu so I go into low availability, and catch up on the weekend – always welcome to try other admins! Naomi is on technical too :)

            Re: Defederation / allowlist: I’m quite sure a lot of instances defederated actually! I believe I read a note on this on a github discussion, or on a lemmy.ml post. IMO if there’s an active security hole which appears to be spreadable to other instances via federation feeds, there is no reasonable basis on which we can decide to trust external instances. Without a patch, they can become infected at any time. I could be missing an angle, but I still feel full defederation was the right call. Happy to hear the arguments against!

            • zen@monyet.cc
              2·
              1 year ago

              don’t think I’ll argue against that, but full defederation requires a restart to bring back the http routes required for federation (or they’ll go 404 instead like what happened this morning), so you’ll have to remember to do that (disable federation while awaiting the security fix, upgrade the server when that fix is out, then enable federation and restart it again to really bring back federation once you’re satisfied that the upgrade is secure).

              maybe if there wasn’t a restart in between the disable and enable federation steps, the http routes wouldn’t need to be re-initialized, but in the case of a security problem, you’ll always need a restart to patch it…

  • ruk_n_rul@monyet.cc
    01·
    1 year ago

    Ah, didn’t realize there’s a site sticky. Sorry about the other post. Everyone pening dealing with this ig. Sucks to not be on PC.

    Still not sure if comments loaded from other instances with custom emoji (the vector of this exploit) can trigger the exploit here, but since we defederated there shouldn’t be a way for it to get in, I hope.