Something unexpected happened as the Lord was preaching the Word in a crowded house in Capernaum (Mark. 2:2). There was noise above them. People on the roof began digging out the topcoat of clay, removing and tearing away clay tiles, then pulling, tugging, and pushing at the underlying branches and timbers. One can surmise that debris from the roof rained down on those inside the house, and they clamored and struggled to get out of the way.
We imagine that bright sunlight suddenly streamed into the darkness of the home from a hole in the roof. But the men above kept working until they made a larger opening. Now particles of dust probably hung in the air from the partial demolition of the roof. Then the stream of light was blocked as a stretcher was swung over the hole, and a man who was a paralytic was lowered into the house (2:4). People shuffled and made room as the man on the stretcher was lowered and placed directly before the Lord. Above the scene now were seen four happy, hopeful friends looking down into the house.
These four determined men are examples of true friends who, in love, help needy sinners to come before the Savior. These four had a faith that refused to quit in the face of obstacles. Their friend couldn’t walk, so they carried him. The crowd blocked their access to Christ, so they went around them. The roof was in the way, so they tore a hole in it. The men went to extraordinary lengths and great effort to place their friend before the Lord.
The paralytic is a picture of all who are outside of Christ for, in our sins, we are all spiritual paralytics. The four men believed in Christ (v. 5), and they knew He could help their friend, so they brought him to Christ and were willing to do whatever it took to get that man before Him. May their same heart beat within us, having the kind of faith and zeal that would tear apart a roof to get a person before the Lord and under the hearing of the gospel. By far, the number one reason new people come to a church or go to a Bible study is that a friend invites them. When you stop and think about that hole in that broken roof, what you see is a picture of love.
Two Minutes with the Bible lets you start your day with short but powerful Bible study articles from the Berean Bible Society. Sign up now to receive Two Minutes With the Bible every day in your email inbox. We will never share your personal information and you can unsubscribe at any time.
“Many people believe that Saul, after his vision on the road to Damascus, began to preach Body truth. But in Galatians 1:23 the Scriptures clearly state that Saul now preached the faith which he once destroyed (Peter’s message).”
As we know, Paul spent the first two chapters in Galatians distinguishing his apostleship and message from the apostleship and message of the Twelve. In fact, he clearly states that the gospel he preached was “not after man,” that is, the Twelve or any other man for that matter. Rather, his message was a direct revelation from the Lord of glory (Gal. 1:11-12). With this in mind, verse 23 must be interpreted in light of the context that precedes it.
“But they [churches of Judea] had heard only, that he which persecuted us in times past now preacheth the faith which once he destroyed. And they glorified God in me” (Gal. 1:23-24).
We must remember that there was a point in Paul’s life when he rejected Christ and the seemingly ridiculous notion that He had risen from the dead. He believed Him to be nothing more than an imposter. But when the Lord of glory appeared to Paul on the dusty road leading to Damascus, his stony heart of flesh melted within him. He could not deny his senses; he believed Christ was Who He claimed to be! You see, the fact that Christ was the Son of God Who had risen from the dead was common ground between the two programs of God. So it is in this sense the apostle preached the faith that he once sought to destroy.
This is further confirmed by Paul’s statement in 1 Corinthians 15:11: “Therefore whether it were I [the apostle of the Gentiles] or they [the twelve apostles of the kingdom], so we preach, and so ye believed.” Preached what? Clearly in this context the resurrection of Christ—true, it may have been for completely different purposes; nevertheless, it was a common denominator.
Two Minutes with the Bible lets you start your day with short but powerful Bible study articles from the Berean Bible Society. Sign up now to receive Two Minutes With the Bible every day in your email inbox. We will never share your personal information and you can unsubscribe at any time.
Robert Gorham Davis was a professor of English at Columbia University. After his retirement he spent his energies writing letters to the New York Times Editorial Section. He wrote many letters, usually prompted by any favorable mention of Christian truth. Anytime the New York Times made some mention of Christianity, he protested with a letter.
In one such letter he wrote, “Plainly if there were such a God who really wished to reveal Himself to mankind, He could do so in a way that left no doubt.”
Well, He did. God revealed Himself to mankind through His Son, and God left no doubt as to the revelation of Himself to mankind by His Son rising from the dead. The resurrection of Jesus Christ is the single most amazing way God could have revealed Himself to us. And the facts are clear, the evidence is conclusive: Christ is risen from the dead.
Satan hates the Resurrection. He tried to cover it up by the lies and scheming of religious leaders and Roman soldiers, and he still tries to cover up the resurrection today. As a result, the true meaning of Easter is covered up with bunnies, eggs, candy, and baskets. But we must be careful and vigilant not to allow the world to cover up for us what this blessed holiday is all about.
Sure, Sealed, and Set
“Now when they were going, behold, some of the watch came into the city, and shewed unto the chief priests all the things that were done” (Matt. 28:11).
Matthew records, “Now when they were going.” The “they” are the women who had come to the tomb on Resurrection morning. An angel had appeared to them at the tomb and told them to tell the disciples that Christ was risen from the dead as He said, and that He was going before them into Galilee where they would see Him (vv. 5-8). As they went from the tomb, the Lord appeared to the women. And then He too told them to tell the disciples that He was risen and would meet them in Galilee.
Thus, verse 11 says, “Now when they were going,” meaning that these women were obeying the Lord and were on their way to deliver to the disciples this message of good news from the risen Savior.
People were moving in different directions on Resurrection morning. The women were on the move toward the disciples’ location, and “some of the watch,” or the Roman soldiers, were on the move as well “into the city” of Jerusalem to find the Jewish chief priests. Strange, isn’t it, that Roman soldiers would seek out the Jewish chief priests?
To answer that question, first we need to learn about the orders given to these soldiers.
“Now the next day, that followed the day of the preparation, the chief priests and Pharisees came together unto Pilate, “Saying, Sir, we remember that that deceiver said, while He was yet alive, After three days I will rise again. “Command therefore that the sepulchre be made sure until the third day, lest His disciples come by night, and steal Him away, and say unto the people, He is risen from the dead: so the last error shall be worse than the first. “Pilate said unto them, Ye have a watch: go your way, make it as sure as ye can. “So they went, and made the sepulchre sure, sealing the stone, and setting a watch” (Matt. 27:62-66).
After Christ’s death on the cross, the chief priests and Pharisees remembered Christ’s words that He would rise again after three days.
They requested from the Roman governor, Pilate, a watch, or a Roman guard unit, to be stationed at the tomb to ensure that the disciples didn’t try to steal Christ’s body and claim He rose again. Pilate granted the watch. So the Roman soldiers “went, and made the sepulchre sure, sealing the stone, and setting a watch.”
Now, Roman guards were serious about what they were put in charge of guarding, because if a Roman soldier failed in his duty as a guard, it was an offense punishable by death. They faced certain and immediate execution. This was why the Philippian jailor was about to kill himself after the earthquake when he thought Paul and all the prisoners had escaped (Acts 16:26-29).
The punishment for failing on guard duty was for the soldier to be “stripped of his clothes, then burned alive in a fire started with the garments.” The fear of that punishment made guards very alert and mindful of their responsibilities.
Christian apologist, evangelist, and author Josh McDowell wrote this about a Roman watch: “A Roman guard unit was a 4-to-16-man security force. Each man was trained to protect six feet of ground. The 16 men in a square of four on each side were supposed to be able to protect 36 yards against an entire battalion and hold it. Normally what they did was this: four men were placed immediately in front of what they were to protect. The other 12 were asleep in a semi-circle in front of them with their heads pointing in. To steal what these guards were protecting, thieves would first have to walk over those who were asleep. Every four hours, another unit of four was awakened and those who had been awake went to sleep. They would rotate this way around the clock.”
We know the guard unit in front of Christ’s tomb was a multi-man force because Matthew recorded how “SOME of the watch came into the city” (Matt. 28:11) after the resurrection. “Some” implies several men of the Roman guard unit.
Matthew 27:66 tells us that the Roman guards “made the sepulchre sure, sealing the stone.” Application of the Roman seal to the stone was key because it was a public testimony that Christ’s body was actually there. Before that seal was applied, each member of the guard unit would go in and verify that what they were guarding was there. Their lives depended on that. That seal tells us that Christ’s body was in that tomb.
After verifying that Christ’s body was in the tomb, the stone was rolled in place, and a cord was stretched across the rock covering the entrance. The cord was fastened at each side with sealing clay. While still soft, the clay packs were stamped with the official signet of the Roman governor.
Trying to move the stone from the tomb’s entrance would have broken the Roman seal, and thus incurred the judgment of Roman law and power. And the consequences were severe, that is, “automatic execution by crucifixion upside down.”
After inspecting the tomb, rolling the stone into place at the entrance, and sealing the stone, the Roman guards then set the watch and got themselves in place.
A Powerful Angel
“And, behold, there was a great earthquake: for the angel of the Lord descended from heaven, and came and rolled back the stone from the door, and sat upon it. “His countenance was like lightning, and his raiment white as snow” (Matt. 28:2-3).
When Christ died, there was a great earthquake. His death in payment for sin was of such magnitude that it shook the world (Matt. 27:50-51). In Matthew 28:2, we see that when He rose again in victory over death, the chains of death were broken forever, and this too was of such magnitude that it literally shook the world. His death shook the world. His resurrection shook the world. Verse 2 calls it “a great earthquake.” The word “great” is megas in the Greek, so we would say that it was a mega-quake. That is to say, it was a supernatural earthquake, and a severe one.
It wasn’t any shifting of the tectonic plates in the earth’s crust that caused the earthquake. It wasn’t any physical phenomenon at all. It was the result of the arrival of the angel of the Lord, descending from heaven to roll the stone away. The earthquake was not caused by Christ coming out of the tomb. It was caused by the arrival of the angel coming to the already-empty tomb. The angel did not come to let Christ out. When the angel arrived, Christ was already gone. He had already risen. In Christ’s resurrected, glorified body, He could simply move right through the rock, which He did when He left the tomb.
We learn that the angel didn’t move the stone to let Christ out. The angel moved the stone to let the world in, so all could see that Christ was gone. The angel moved the stone so that the women and the apostles could go in and give eyewitness testimony to the fact that Christ wasn’t there.
You gotta love this angel. When he comes, he comes in power, and there’s a great earthquake. He then single-handedly rolls that great stone some distance from the entrance of the tomb, and then he sits on it. That stone being removed from that door showed death had been conquered forever, that we are free forever from bondage through the fear of death (Heb. 2:15). That stone was a symbol of the sin that shut man in prison and condemned him to death. But now Christ had conquered sin and the grave. Nothing was going to change this fact. And in exultation at Christ’s victory, this angel sat on that stone. He sat on it as a divine testimony to Christ’s once-for-all, finished sacrifice for sin and His triumphant resurrection over death.
Shaky Soldiers
“And for fear of him the keepers did shake, and became as dead men” (Matt. 28:4).
When the angel appeared and the earthquake occurred, and he moved the stone and sat on it, Matthew 28:4 records that the guards shook. The Greek word translated “shake” (seio) comes from the Greek word for “earthquake” in verse 2 (seismos). One could say that the guards had their own seismic experience. The ground quaked and then they quaked and shook out of sheer terror from seeing this heavenly angel.
Roman soldiers were acquainted with the terrors of battle. They were not easily shaken emotionally. But the angel’s shining countenance, coupled with the earthquake, paralyzed the “keepers,” the Roman guard. They were struck with such fear that they went into a comatose state, and they fainted “and became as dead men.” The angel never speaks to these guards. It was just his powerful presence that overwhelmed these rough and tough Roman legionnaires.
As soon as they regained consciousness, the soldiers checked the tomb, which they found empty except for the graveclothes lying as they were when they were wrapped around the Lord’s body, and the head napkin folded neatly to the side (Jn. 20:5-7). The soldiers saw and examined the entire scene.
“Now when they were going, behold, some of the watch came into the city, and shewed unto the chief priests all the things that were done” (Matt. 28:11).
Then some of the guards came into the city of Jerusalem to talk to the chief priests and to report to them all that had happened. Only “some” of them came because the others probably feared for their lives and were in hiding. The “some” who did come were shrewd. They did not report to their superior officers or to Governor Pilate; instead they reported to the Jewish chief priests. They reported to the men who were as anxious as they were to cover up what had happened.
They reported to “the chief priests all the things that were done.” And “all things that were done” included the earthquake, the appearance of the angel, the angel moving the stone away from the entrance to the tomb by himself and then sitting on it, and how, after this, they fainted. Then they reported that, after they came to, the tomb was empty, except for the graveclothes. The chief priests, therefore, received word of Christ’s resurrection before the disciples did. The religious leaders already knew while the women were still on their way to the disciples.
A Bribe and a Lie
“And when they were assembled with the elders, and had taken counsel, they gave large money unto the soldiers, “Saying, Say ye, His disciples came by night, and stole Him away while we slept. “And if this come to the governor’s ears, we will persuade him, and secure you” (Matt. 28:12-14).
In verse 12 we read, “when they [the chief priests] were assembled with the elders, and had taken counsel.” The chief priests immediately called for a formal, emergency meeting of the Sanhedrin, the 70-man ruling body of the Jewish religious leaders. The firsthand facts from the soldiers were shared: the tomb is empty and they’ve got to do something about it. They assembled to consult together, that together they might come to a satisfactory resolution.
First they decided to bribe the soldiers to lie: “they gave large money unto the soldiers” (v. 12). They couldn’t have the soldiers going around saying there was an earthquake, an angel who rolled the stone away, and then that the tomb was empty. Today we would call this hush money.
The Greek word for money in verse 12 means silver coinage. The Jewish leaders gave the soldiers a large sum of silver money. They had bought Judas off for the sum of 30 pieces of silver; now they had to pay much more, but there was no price too high to buy a lie about this. This information could not come out. One would think that the resurrection would soften the hearts of the religious leaders to believe, but instead it hardened their hearts. They were determined in their unbelief.
Second, the Jewish religious leaders decided that, in return for the large sum of money, the soldiers were to spread a lie: “Say ye, His disciples came by night, and stole Him away while we slept” (v. 13). These religious leaders of Israel were desperate to come up with a story to explain the empty tomb and deter people from faith in Christ. The body was stolen. This was the only lie that worked. Nothing else works but this, because you have to offer a natural explanation for an empty tomb. But their cover-up proves the resurrection of Jesus Christ.
The lie to be made known was “Well, we were all asleep when the disciples came in the middle of the night and stole His body.” However, the question that logically raises is “If you were asleep, how do you know His disciples came during the night and stole His body away?” You can’t have it both ways. You can’t be asleep and also know who took Him.
Another reason for the large bribe was that the religious leaders were asking the soldiers to incriminate themselves and to admit to sleeping while on guard duty in violation of Roman military law, which, as previously stated, was punishable by death.
For this reason, the third thing the religious ruling body decided was to assure the soldiers of their protection, telling them, “And if this come to the governor’s ears, we will persuade him, and secure you.” They were good at that, persuading Governor Pilate. Pilate was like putty in their hands. When Pilate was resistant to crucifying Christ, they kept working on him and worked the crowd into a frenzy against him: “the Jews cried out, saying, If thou let this Man go, thou art not Caesar’s friend” (Jn. 19:12). As we know, the weak-kneed Pilate ultimately relented and commanded Christ to be crucified. Thus, if Pilate tried to take action to execute the soldiers, the chief priests knew they could go to him and keep the soldiers out of trouble.
“So they took the money, and did as they were taught: and this saying is commonly reported among the Jews until this day” (Matt. 28:15).
Confident that the chief priests could do this and keep them safe, we learn in verse 15 that the soldiers took the large sum of money and did what they had been “taught.” The Jewish leaders had a number of soldiers that all needed to be telling the exact same story. They needed to go over the details of the cover-up and the lie so that one soldier wasn’t saying one thing and another was saying something different. As we say today, they needed to be on the same page. Thus, they were “taught” the lie carefully.
The leaders feared that the Resurrection was going to be preached and believed. They knew the news that the tomb was empty was going to spread, so they had to spread the lie quickly and widely. As Mark Twain once said, “A little lie can travel half way ’round the world while Truth is still lacing up her boots.” The Resurrection cover-up lie of the disciples stealing the body went out far and wide. Matthew, writing years later, confirms that it was still a prominent story believed by the Jews in that day, even though it was ridiculous.
The soldiers, who knew better, lied. The religious leaders, who knew better, lied. Satan, who is the father of lies, still propagates lies about the Resurrection today, and people still believe them.
Here are a few lies about the resurrection. There’s the “Wrong Tomb Theory,” that says the reason the stone was rolled away and the tomb was empty was that the women went to the wrong tomb. However, two of the women had watched where He was buried the day of His crucifixion and knew where the tomb was (Matt. 27:61). This would mean that not only did the women go to the wrong tomb, but so did Peter and John when they ran to the tomb later (Jn. 20:3-4). And the right tomb would have been located after a while, especially by the Lord’s enemies who would have wanted to disprove the resurrection. Obviously, Joseph of Arimathea and the Roman soldiers knew which tomb contained the Lord’s body.
There’s the “Swoon Theory,” that Christ didn’t really die on the cross and just fainted from exhaustion. He was buried alive, and so later He came to and left the tomb on His own.
“A woman wrote [pastor and radio preacher] J. Vernon McGee, ‘Our preacher said that on Easter Jesus just swooned on the cross, and the disciples nursed him back to health. What do you think?’
“McGee replied, ‘Dear Sister, beat your preacher with a leather whip for thirty-nine heavy strokes. Nail him to a cross. Hang him in the sun for six hours. Run a spear through his heart. Embalm him. Put him in an airless tomb for three days. Then see what happens.’”
You may have heard the “Hallucination Theory.” This theory states that people only thought they had seen the resurrected Christ and had just hallucinated. However, mass hallucinations do not exist, and numbers of people together in the same place saw the risen Christ. There were too many different people in too many different places at too many different times and circumstances to be able to have that many hallucinations. And if the risen Christ were a hallucination, there would have been a body in the tomb.
Then there’s the “Stolen Body Theory” of Matthew 28:12-15 that began on Resurrection Day, a theory which is still believed by some. However, the disciples weren’t plotting how to steal the Lord’s body, they were hiding in fear of their lives.
He is Risen Indeed
What we have here in Matthew 28:11-15 is an account of Christ’s resurrection from the viewpoint of His enemies. The Holy Spirit, in the wisdom of God, gives us this angle as proof of the truth. Critics and doubters of the Resurrection may question the accounts by Christ’s friends, saying that of course they’re going to lie and say He rose again, because they were His friends. So unbelievers might dismiss the testimony of His followers. But it’s pretty hard to dismiss the testimony of His enemies! The last thing the Jewish leaders wanted was a resurrection and an empty tomb, but this is exactly what they got, and the elaborate cover-up is proof that the Resurrection really happened.
Christ’s followers give strong testimony to His resurrection in their eyewitness accounts, and His enemies give strong testimony to it in the plot to cover it up. Any and every way you look at it, Jesus Christ rose from the dead.
Praise God that He did! His resurrection is the cornerstone of our faith. The fact that Christ came out of that tomb is an everlasting proof of the sufficiency of His sacrifice for our sins and that our sin debt was paid in full, and that we will spend eternity in the heavenly places. As we are reminded in 2 Corinthians 4:14,
“Knowing that He which raised up the Lord Jesus shall raise up us also by Jesus.”
You can receive More Minutes With the Bible every week in your email inbox. This list features longer articles, including both original content and articles that have appeared in the Berean Searchlight.
I'm home! Well, for a day, then it's off to the other side of the country (which I just flew over last night on the way back from Dublin 🤦♂️) for an event at the Microsoft Accelerator in Perth on Monday. Such is the path we've taken, but it does provide some awesome opportunities to meet up with folks around the world and see some really interesting stuff. Come by if you're over that way or if you're on the east coast of Aus, I'll be at NDC Melbourne only a couple of weeks later. And somewhere in the midst of all that, we'll get this HIBP UX rebuild finished...
After an unusually long day of travelling from Iceland, we've finally made it to the land of Guinness, Leprechauns, and a tax haven for tech companies. This week, there are a few more lessons from the successful phish against me the previous week, and in happier news, there is some really solid progress on the HIBP UX rebuild. We spent a bunch of time with Stefan and Ingiber (the guy rebuilding the front end) whilst in Reykjavik and now have a very clear plan mapped out to get this finished in the next 6 weeks. More on that in this week's update, enjoy!
Well, this certainly isn't what I expected to be talking about this week! But I think the fact it was someone most people didn't expect to be on the receiving end of an attack like this makes it all the more consumable. I saw a lot of "if it can happen to Troy, it can happen to anyone" sort of commentary and whilst it feels a bit of obnoxious for me to be saying it that way, I appreciate the sentiment and the awareness it drives. It sucked, but I'm going to make damn sure we get a lot of mileage out of this incident as an industry. I've no doubt whatsoever this is a net-positive event that will do way more good than harm. On that note, stay tuned for the promised "Passkeys for Normal People" blog post, I hope to be talking about that in next week's video (travel schedule permitting). For now, here's the full rundown of how I got phished:
You know when you're really jet lagged and really tired and the cogs in your head are just moving that little bit too slow? That's me right now, and the penny has just dropped that a Mailchimp phish has grabbed my credentials, logged into my account and exported the mailing list for this blog. I'm deliberately keeping this post very succinct to ensure the message goes out to my impacted subscribers ASAP, then I'll update the post with more details. But as a quick summary, I woke up in London this morning to the following:
I went to the link which is on mailchimp-sso.com and entered my credentials which - crucially - did not auto-complete from 1Password. I then entered the OTP and the page hung. Moments later, the penny dropped, and I logged onto the official website, which Mailchimp confirmed via a notification email which showed my London IP address:
I immediately changed my password, but not before I got an alert about my mailing list being exported from an IP address in New York:
And, moments after that, the login alert from the same IP:
This was obviously highly automated and designed to immediately export the list before the victim could take preventative measures.
There are approximately 16k records in that export containing info Mailchimp automatically collects and they appear as follows:
Every active subscriber on my list will shortly receive an email notification by virtue of this blog post going out. Unfortunately, the export also includes people who've unsubscribed (why does Mailchimp keep these?!) so I'll need to work out how to handle those ones separately. I've been in touch with Mailchimp but don't have a reply yet, I'll update this post with more info when I have it.
I'm enormously frustrated with myself for having fallen for this, and I apologise to anyone on that list. Obviously, watch out for spam or further phishes and check back here or via the social channels in the nav bar above for more. Ironically, I'm in London visiting government partners, and I spent a couple of hours with the National Cyber Security Centre yesterday talking about how we can better promote passkeys, in part due to their phishing-resistant nature. 🤦♂️
More soon, I've hit the publish button on this 34 mins after the time stamp in that first email above.
More Stuff From After Initial Publish
Every Monday morning when I'm at home, I head into a radio studio and do a segment on scams. It's consumer-facing so we're talking to the "normies" and whenever someone calls in and talks about being caught in the scam, the sentiment is the same: "I feel so stupid". That, friends, is me right now. Beyond acknowledging my own foolishness, let me proceed with some more thoughts:
Firstly, I've received a gazillion similar phishes before that I've identified early, so what was different about this one? Tiredness, was a major factor. I wasn't alert enough, and I didn't properly think through what I was doing. The attacker had no way of knowing that (I don't have any reason to suspect this was targeted specifically at me), but we all have moments of weakness and if the phish times just perfectly with that, well, here we are.
Secondly, reading it again now, that's a very well-crafted phish. It socially engineered me into believing I wouldn't be able to send out my newsletter so it triggered "fear", but it wasn't all bells and whistles about something terrible happening if I didn't take immediate action. It created just the right amount of urgency without being over the top.
Thirdly, the thing that should have saved my bacon was the credentials not auto-filling from 1Password, so why didn't I stop there? Because that's not unusual. There are so many services where you've registered on one domain (and that address is stored in 1Password), then you legitimately log on to a different domain. For example, here's my Qantas entry:
And the final thought for now is more a frustration that Mailchimp didn't automatically delete the data of people who unsubscribed. There are 7,535 email addresses on that list which is nearly half of all addresses in that export. I need to go through the account settings and see if this was simply a setting I hadn't toggled or something similar, but the inclusion of those addresses was obviously completely unnecessary. I also don't know why IP addresses were captured or how the lat and long is calculated but given I've never seen a prompt for access to the GPS, I imagine it's probably derived from the IP.
I'll park this here and do a deeper technical dive later today that addresses some of the issues I've raised above.
The Technical Bits
I'll keep writing this bit by bit (you may see it appear partly finished while reading, so give the page a refresh later on), starting with the API key that was created:
This has now been deleted so along with rolling the password, there should no longer be any persistent access to the account.
By no means would I encourage people not to enable 2FA via OTP, but let this be a lesson as to how completely useless it is against an automated phishing attack that can simply relay the OTP as soon as it's entered. On that note, another ridiculous coincidence is that in the same minute that I fell for this attack, I'd taken a screen cap of the WhatsApp message below and shown Charlotte - "See, this reinforces what we were talking about with the NCSC yesterday about the importance of passkeys":
Another interesting angle to this is the address the phish was sent to:
The rest of that address is probably pretty predictable (and I do publish my full "normal" address on the contact page of this blog, so it's not like I conceal it from the public), but I find it interesting that the phish came to an address only used for Mailchimp. Which leaves two possibilities:
Someone specifically targeted me and knew in advance the pattern I use for the address I sign up to services with. They got it right first go without any mail going to other addresses.
Someone got the address from somewhere else, and I've only ever used it in one place...
Applying some Occam's razor, it's the latter. I find the former highly unlikely, and I'd be very interested to hear from anyone else who uses Mailchimp and received one of these phishes.
Still on email addresses, I originally read the phish on my iThing and Outlook rendered it as you see in the image above. At this point, I was already on the hook as I intended to login and restore my account, so the way the address then rendered on the PC didn't really stand out to me when I switched devices:
Curious as to why unsubscribed users were in the corpus of exported data, I went searching for answers. At no point does Mailchimp's page on unsubscribing mention anything about not deleting the user's data when they opt out of receiving future emails. Keeping in mind that this is AI-generated, Google provided the following overview:
That "Purpose of Keeping Unsubscribes" section feels particularly icky and again, this is the AI and not Mailchimp's words, but it seems to be on point. I can go through and delete unsubscribed addresses (and I'll do that shortly as the last thing I'm going to do now is rush into something else), but then it looks like that has to be a regular process. This is a massive blindspot on Mailchimp's behalf IMHO and I'm going to provide that feedback to them directly (just remembered I do know some folks there).
I just went to go and check on the phishing site with the expectation of submitting it to Google Safe Browsing, but it looks like that will no longer be necessary:
2 hours and 15 minutes after it snared my creds, Cloudflare has killed the site. I did see a Cloudflare anti-automation widget on the phishing page when it first loaded and later wondered if that was fake or they were genuinely fronting the page, but I guess that question is now answered. I know there'll be calls of "why didn't Cloudflare block this when it was first set up", but I maintain (as I have before in their defence), that it's enormously hard to do that based on domain or page structure alone without creating a heap of false positives.
On the question of the lat and long in the data, I just grabbed my own records and found an IP address belonging to my cellular telco. I had two records (I use them to test both the daily and weekly posts), both with the same IP address and created within a minute of each other. One had a geolocation in Brisbane and the other in far north Queensland, about 1,700km away. In other words, the coords do not pinpoint the location of the subscriber, but the record does contain "australia/brisbane,au,qld" so there's some rough geolocation data in there.
Loading the List into Have I Been Pwned
When I have conversations with breached companies, my messaging is crystal clear: be transparent and expeditious in your reporting of the incident and prioritise communicating with your customers. Me doing anything less than that would be hypocritical, including how I then handle the data from the breach, namely adding it to HIBP. As such, I’ve now loaded the breach and notifications are going out to 6.6k impacted individual subscribers and another 2.4k monitoring domains with impacted email addresses.
Looking for silver linings in the incident, I’m sure I’ll refer this blog post to organisations I disclose future breaches to. I’ll point out in advance that even though the data is “just” email addresses and the risk to individuals doesn’t present a likelihood of serious harm or risk their rights and freedoms (read that blog post for more), it’s simply the right thing to do. In short, for those who read this in future, do not just as I say, but as I do.
The Washup
I emailed a couple of contacts at Mailchimp earlier today and put two questions to them:
Are passkeys on your roadmap
Where does Mailchimp stand on “unsubscribe” not deleting the data
A number of people have commented on social media about the second point possibly being to ensure that someone who unsubscribes can’t then later be resubscribed. I’m not sure that argument makes a lot of sense, but I’d like to see people at least being given the choice. I’m going to wait on their feedback before deciding if I should delete all the unsubscribed emails myself, I’m not even sure if that’s possible via the UI or requires scripting against the API,.
The irony of the timing with this happening just as I’ve been having passkey discussions with the NCSC is something I’m going to treat as an opportunity. Right before this incident, I’d already decided to write a blog post for the normies about passkey, and now I have the perfect example of their value. I’d also discussed with the NCSC about creating a passkey equivalent of my whynohttps.com project which highlighted the largest services not implementing HTTPS by default. As such, I’ve just registered whynopasskeys.com (and its singular equivalent) and will start thinking more about how to build that out so we can collectively put some pressure on the services that don’t support unphishable second factors. I actually attempted to register that domain whilst out walking today, only to be met with the following courtesy of DNSimple:
Using a U2F key on really important stuff (like my domain registrar) highlights the value of this form of auth. Today’s phish could not have happened against this account, nor the other critical ones using a phishing resistant second factor and we need to collectively push orgs in this direction.
Sincere apologies to anyone impacted by this, but on balance I think this will do more good than harm and I encourage everyone to share this experience broadly.
Update 1: I'll keep adding more thoughts here via updates, especially if there's good feedback or questions from the community. One thing I'd intended to add earlier is that the more I ponder this, the more likely I think it is that my unique Mailchimp address was obtained from somewhere as opposed to guessed in any targeted fashion. A possible explanation is the security incident they had in 2022, which largely targeted crypto-related lists, but I imagine would likely have provided access to the email addresses of many more customers too. I'll put that to them when I get a response to my earlier email.
Update 2: I now have an open case with Mailchimp and they've advised that "login and sending for the account have been disabled to help prevent unauthorized use of the account during our investigation". I suspect this explains why some people are unable to now sign up to the newsletter, I'll try and get that reinstated ASAP (I'd rolled creds immediately and let's face it, the horse has already bolted).
Pondering this even further, I wonder if Mailchimp has any anti-automation controls on login? The credentials I entered into the phishing site were obviously automatically replayed to the legitimate site, which suggests something there is lacking.
I also realised another factor that pre-conditioned me to enter credentials into what I thought was Mailchimp is their very short-lived authentication sessions. Every time I go back to the site, I need to re-authenticate and whilst the blame still clearly lies with me, I'm used to logging back in on every visit. Keeping a trusted device auth'd for a longer period would likely have raised a flag on my return to the site if I wasn't still logged in.
Update 3: Mailchimp has now restored access to my account and the newsletter subscription service is working again. Here's what they've said:
We have reviewed the activity and have come to the same conclusion that the unauthorized export and API key from 198.44.136.84 was the scope of the access. Given we know how the access took place, the API key has been deleted, and the password has been reset, we have restored your access to the account.
They've also acknowledged several outstanding questions I have (such as whether passkeys are on the roadmap) and have passed them along to the relevant party. I'll update this post once I have answers.
There's been a lot of discussion around "Mailchimp are violating my local privacy laws by not deleting emails when I unsubscribe", and that's one of the outstanding questions I've sent them. But on that, I've had several people contact me and point out this is not the case as the address needs to be retained in order to ensure an opted-out individual isn't later emailed if their address is imported from another source. Read this explainer from the UK's ICO on suppression lists, in particular this para:
Because we don’t consider that a suppression list is used for direct marketing purposes, there is no automatic right for people to have their information on such a list deleted.
I suspect this explains Mailchimp's position, but I suggest that should be clearer during the unsubscribe process. I just went through and tested it and at no time is it clear the email address will be retained for the purpose of supression:
My suggestion would be to follow our approach for Have I Been Pwned where we give people three choices and allow them to choose how they'd like their data to be handled:
At present, Mailchimp is effectively implementing the first option we provide and the folks that are upset were expecting the last option. Hopefully they'll consider a more self-empowering approach to how people's data is handled, I'll update this blog post once I have their response.
Update 4: Someone has pointed out that the sending email address in the phish actually belongs to a Belgian cleaning company called Group-f. It's not unusual for addresses like this to be used to send malicious mail as they usually don't have a negative reputation and more easily pass through spam filters. It also indicates a possible compromise on their end, so I've now reached to them to report the incident.
Update 5: I've been contacted by someone that runs a well-known website that received the same phishing email as me. They made the following observation regarding the address that received the phish:
We have subscribed to Mailchimp with an address that is only used to subscribe to services, no outgoing communication from us. The phishing emails were delivered to exactly this address, couldn't yet find them on any other address. This makes me very much believe that possibility #2 is the case - they got the address from somewhere.
This aligns with my earlier observation that a customer list may have been obtained from Mailchimp and used to send the phishing emails. They went on to say they were seeing multiple subsequent phishes targeting their Mailchimp account.
Btw, we got some more (Mailchimp) phishing emails today — same style, this time 4 times writing about a new login detected, and once that an abuse report was received and we needed to take immediate action.
That a customer list may have been compromised was one of the questions I put to Mailchimp and am still awaiting an answer on. That was about 36 hours ago now, so I've just given them a little nudge.
Update 6: There have been a lot of suggestions that Mailchimp should be storing the hashes of unsubscribed emails rather than the full addresses in the clear. I understand the sentiment, and it does offer some protection, but it by no means ticks the "we no longer have the address" box. This is merely pseudoanonymisation, and the hashed address can be resolved back to the clear if you have a list of plain text candidates to hash and compare them to. There's a good explainer of this in the answer to this question on Security Stack Exchange about hashing email addresses for GDPR compliance. IMHO, my example of how we handle this in HIBP is the gold standard that Mailchimp should be implementing.
And there's also another problem: short of cracking the hashed addresses, you can never export a list of unsubscribed email addresses, for example, if you wanted to change mail campaign provider. The only way that would work is if the hashing algorithm is the same in the destination service, or you build some other level of abstraction at any other future point where you need to compare plain text values to the hashed impression list. It's messy, very messy.
Update 7: Validin has written a fantastic piece about Pulling the Threads of the Phish of Troy Hunt that takes a deep dive into the relationship between the domain the phish was hosted on and various other campaigns they've observed.
Given these similarities, we believe the phishing attempt of Troy Hunt is very likely Scattered Spider.
Scattered Spider certainly has previous form, and this was a very well-orchestrated phish. Four days on as I write this, it's hard not to be a bit impressed about how slick the whole thing was.
It's time to fly! 🇬🇧 🇮🇸 🇮🇪 That's two new flags (or if you're on Windows and can't see flag emojis, that's two new ISO codes) I'll be adding to my "places I've been list" as we start the journey by jetting out to London right after I publish this blog. If you're in the area, I'll be speaking at Oxford University on Wednesday at 17:00 and that's a free and open event. And since recording this morning, we have managed to confirm that I will be speaking at a community event in Reykjavik the following Monday morning, and you'll see a link on my 2025 events page as soon as they make one available. No public events planned for Ireland yet, but if you're in Dublin and would like to run something the week after I'm in Iceland, get in touch. Just to round out a big schedule, I'll be back in Aus speaking in Perth at Microsoft's Student Accelerator on 14 April and then it's off to NDC Melbourne shortly after that for a talk on the 30th. Then rest 🙂
What an awesome response to the new brand! I'm so, so happy with all the feedback, and I've gotta be honest, I was nervous about how it would be received. The only negative theme that came through at all was our use of Sticker Mule, which apparently is akin to being a Tesla owner. Political controversy aside, this has been an extremely well-received launch and I've also loved seeing the issues raised on the open source repo for the front end and Ingiber's (near instant!) addressing of each and every one of them. Please keep that feedback coming, and I'll talk more about some of the changes we've made as a result in the next weekly update.
Designing the first logo for Have I Been Pwned was easy: I took a SQL injection pattern, wrote "have i been pwned?" after it and then, just to give it a touch of class, put a rectangle with rounded corners around it:
The rebrand we're soft-launching today has been a long time coming, and true to that form, we're not rushing it. This is a "soft launch" in that we're sharing work in progress that's sufficiently evolved to put it out there to the public, but you won't see it in production anywhere yet. The website is no different, the social channels still have the same hero shots and avatars etc. This is the time to seek feedback and tweak before committing more effort into writing code and pushing this to the masses.
A quick primer on "why", as the question has come up a few times whilst previously discussing this. Assume for a moment that my valiant 2013 attempt at a logo was, itself, aesthetically sufficient. It's a hard one to use in different use cases (favicon, merch) and it's quite "busy" in it's current form with no easily recognisable symbol which makes it hard to apply to many use cases. And there are loads of use cases; I mentioned a couple just now, but how about in formal documents such a the contracts we write for enterprise customers? Or as it appears on Stripe-generated invoices, stickers, my 3D printed logos, email signatures and so on and so forth. And branding isn't just a logo, it's a whole set of different use cases and variants of the logo and colours such that you have flexibility to present the brand's image in a cohesive, recognisable fashion. Branding is an art form.
At one point there, I'd had a go at redoing the logo myself. It was terrible. You know how you can have this vision of something aesthetic in your mind and know instantly if it's the right thing when you see it, but just can't quite articulate it yourself? I'm like that with interior design... and logos. So, I reached out to Fiverr for help, and immediately regretted it:
I mean... wow. Ok, I get free revisions, let's give the designer another chance:
Dammit! This just wasn't going to work, and we were going to need to make a much more serious commitment if we wanted this done right. So, we went to Luft Design in Norway as Charlotte and Mikael went way back, and with his help, we went around and around through various iterations of mood boards, design styles, colours and carved out time in Oslo during our visit there in December to sit with Stefan as well and really nut this thing out. I was adamant that I wanted something immediately recognisable but also modern and cohesive without being fussy. Basically, give me everything, which Mikael did:
Let me talk you through the logic of these three variations, beginning with the icon. Mikael initially gave us multiple possible variations of a totally different icons which implied different things. My issue with that is you have to know what the symbology means in order for it to make sense. Perhaps if you're starting from scratch that can work, but when you're a decade+ into a name and a brand, there's history that I think you need to carry forward. One of the variations Mikael did reused that original SQL injection pattern I applied to the logo back in 2013 and just for the sake of justifying my choice, here's what it means for the uninitiated:
Take a SQL query like this:
SELECT * From User WHERE Name = 'blah'
Now, imagine "blah" is untrusted user input, that being data that someone submits via a form, for example. They might then change "blah" to the following:
blah';DROP TABLE USER
We'll shortcut the whole SQL injection lesson about validation of untrusted data and parameterization of queries and just jump straight to the resultant query:
SELECT * From User WHERE Name = 'blah';DROP TABLE USER'
And now, due to the additional query appended to the original one, your user table is gone. However... the SQL has a syntax error as there's a rogue apostrophe hanging off the end, so we fix it by using commenting syntax like so:
blah';DROP TABLE USER;--
Chief among the characters in that pattern are these guys:
';--
And that's the history; these are characters that play a role in the form of attack that has led to so many of the breaches in HIBP today. Turns out they're also really easy to stylise and represent as a concise logo:
We agonised over variations of this for months. The problem is that when you think about all the ones that are really recognisable without accompanying words, they're recognisable because the brand is massive. The Nike swoosh, the Mitsubishi diamonds, the Pepsi circle, the Apple logo etc. HIBP obviously doesn't have that level of cachet, but I really like the simplicity of reach of those, and that's what we have with this one as well as that connection to the history of the brand and the practical use of those characters.
But just as with many of those other recognisable logos, these are times when what is effectively just a logo alone isn't enough, so we have the longer form version:
"Have I Been Pwned" is a mouthful. It's not just long to say, it's long to put on the screen, long to print as a sticker, long to put on a shirt and so on and so forth. "Pwned", on the other hand, is short, concise and, I'd argue, has acheived much greater recognition as a word due to HIBP. Reading how “PWNED” went from hacker slang to the internet’s favorite taunt, I think that's a fair conclusion to draw. For a moment, we even toyed with the idea of an actual rename to just "Pwned" and looked at trying to buy pwned.com via a broker which, uh, didn't work out real well:
Appartently, you can put a price on it! So no, we're not renaming anything, we're just providing various stylistic options for representing the logo. This is why we still have the much wordier versions as well:
Unlike old mate at Fiverr, a proper branding exercise like Mikael has done goes well beyond just the logo alone. For example, we have a colour palette:
And we have typography:
Hoodies:
And t-shirts:
You get the idea.
But most importantly, there's the website. Obviously the brand needs to prevail across to the digital realm, but there's also the issue of the front-end tech stack we build on, and that's something I've been thinking about for months now:
In 2013, I built the front end of @haveibeenpwned on Bootstrap and jQuery. In 2025, @stebets and I are rebuilding it as part of a rebrand. What should we use? What are the front end tools that make web dev awesome today? (vanilla HTML, CSS and JS aside, of course)
You can read all sorts of different suggestions in that thread but in the end, we decided to keep it simple:
Bootstrap 5
Vanilla JS (i.e. just write JavaScript without a framework dependency like jQuery)
Sass (which compiles to CSS anyway)
And that's it. Except Stefan and I are busy guys and we really didn't want to invest our precious cycles rebuilding the front end, so we got Ingiber Olafsson to do it. Ingiber came to us via Stefan (so now we have two Icelanders, two Norwegians and... me), and he's been absolutely smashing out the new front end of HIBP:
What I've really enjoyed with Ingiber's approach is that everything he's built is super clean, lightweight and visually beautiful (based on Mikael's work, of course). I've really appreciated his attention to detail that isn't always obvious too, for example making sure accessibility for the visually impaired is maximised:
Ingiber has helped get us to the point where very soon, Stefan and I will begin the integration work to roll the new brand into the main website. That's not just branding work either as the UX is getting a major overhaul. Some stuff is fairly minor: the list of pwned websites is now way too large and we need to have a dedicated page per breach. Other stuff is much more major: we want to have a specific "login" facility (quoting as it will likely remain passwordless by sending a token via email), where we'll then consolidate everything from notification enrolment to domain management to viewing stealer logs. It's a significant paradigm shift that requires a lot of very careful thought.
A quick caveat on the examples above and the others in the repository: we've given Ingiber free reign to experiment and throw ideas around. As a result, we've got some awsome stuff we hadn't thought about before. We've also got some stuff that will be infeasible in the short term, for example, a link through to the official response of the breached company and the full timeline of events. I hope ideas like this keep coming (both from Ingiber and the community), but just keep in mind that some things you see in this repo won't be on the website the day we roll all this out.
As with so much of this project since day one, we're doing this out in the open for everyone to see. Part of that is this blog post heralding what's to come, and part of it is also open sourcing the ux-rebuild repository. I actually created that repo more than a year ago and started crowd-sourcing ideas before closing it off last month whilst Ingiber got working. It's now open again, and I'd like to invite anyone interested to check out what we're building, leave their comments (either here on in the repo), send PRs and so on and so forth. I'm really stoked with the work the guys I've mentioned in this blog post have done, but there will be other great ideas that none of us have thought of yet. And if you come up with something awesome, we already have truckloads of stickers and 3D printed logos I'd love to send you:
So there we have it, that's the rebrand. Do please send us your feedback, not just about logos and look and feel, but also what you'd like to see UX and feature wise on the website. The discussions list on that repo is a great place the chime in or add new ideas, or even just the comments section below 👇
Edit: Wow, all the responses have been awesome! Gotta be honest, I was nervous redefining the brand after so long, but I couldn't have hoped for a better response 😊 I have two quick additions to this post:
I should have thought of this before publishing this post, but we've now published the static HTML pages to preview.haveibeenpwned.com. This is running on Cloudflare pages and is auto-deployed on each GitHub merge into main, so you'll see this continue to evolve over the coming weeks.
We survived the cyclone! That was a seriously weird week with lots of build-up to an event that last occurred before I was born. It'd been 50 years since a cyclone came this far south, and the media was full of alarming predictions of destruction. In the end, we maxed out at 52kts just after I recorded this video:
It’s here. But 47kts max gusts isn’t too bad, nothing actually blowing over here (yet). pic.twitter.com/qFyrZdiyRW
We remained completely untouched and unaffected beyond needing to sweep up some leaves once the rain (which has also been unremarkable), finally stops. It appears the worst damage has been a lot of homes without power and perhaps most obviously, the beaches have done a complete vanishing act with all the sand:
What our favourite beach is like today, versus before. They’ll rebuild it, this isn’t unprecedented, but yeah, there’s some work to be done now. pic.twitter.com/6zFMG7bZqK
I think I've finally caught my breath after dealing with those 23 billion rows of stealer logs last week. That was a bit intense, as is usually the way after any large incident goes into HIBP. But the confusing nature of stealer logs coupled with an overtly long blog post explaining them and the conflation of which services needed a subscription versus which were easily accessible by anyone made for a very intense last 6 days. And there were the issues around source data integrity on top of everything else, but I'll come back to that.
When we launched the ability to search through stealer logs last month, that wasn't the first corpus of data from an info stealer we'd loaded, it was just the first time we'd made the website domains they expose searchable. Now that we have an actual model around this, we're going to start going back through those prior incidents and backfilling the new searchable attributes. We've just done that with the 26M unique email address corpus from August last year and added a bunch previously unseen instances of an email address mapped against a website domain. We've also now flagged that incident as "IsStealerLog", so if you're using the API, you'll see that attribute now set to true.
For the most part, that data is all handled just the same as the existing stealer log data: we map email addresses to the domains they've appeared against in the logs then make all that searchable by full email address, email address domain or website domain (read last week's really, really long blog post if you need an explainer on that). But there's one crucial difference that we're applying both to the backfilling and the existing data, and that's related to a bit of cleaning up.
A theme that emerged last week was that there were email addresses that only appeared against one domain, and that was the domain the address itself was on. If john@gmail.com is in there and the only domain he appears against is gmail.com, what's up with that? At face value, John's details have been snared whilst logging on to Gmail, but it doesn't make sense that someone infected with an info stealer only has one website they've logging into captured by the malware. It should be many. This seems to be due to a combination of the source data containing credential stuffing rows (just email and password pairs) amidst info stealer data and somewhere in our processing pipeline, introducing integrity issues due to the odd inputs. Garbage in, garbage out, as they say.
So, we've decided to apply some Occam's razor to the situation and go with the simplest explanation: a single entry for an email address on the domain of that email address is unlikely to indicate an info stealer infection, so we're removing those rows. And not adding any more that meet that criteria. But there's no doubt the email address itself existed in the source; there is no level of integrity issues or parsing errors that causes john@gmail.com to appear out of thin air, so we're not removing the email addresses in the breach, just their mapping to the domain in the stealer log. I'd already explained such a condition in Jan, where there might be an email address in the breach but no corresponding stealer log entry:
The gap is explained by a combination of email addresses that appeared against invalidly formed domains and in some cases, addresses that only appeared with a password and not a domain. Criminals aren't exactly renowned for dumping perfectly formed data sets we can seamlessly work with, and I hope folks that fall into that few percent gap understand this limitation.
FWIW, entries that matched this pattern accounted for 13.6% of all rows in the stealer log table, so this hasn't made a great deal of difference in terms of outright volume.
This takes away a great deal of confusion regarding the infection status of the address owner. As part of this revision, we've updated all the stealer log counts seen on domain search dashboards, so if you're using that feature, you may see the number drop based on the purged data or increase based on the backfilled data. And we're not sending out any additional notifications for backfilled data either; there's a threshold at which comms becomes more noise than signal and I've a strong suspicion that's how it would be received if we started sending emails saying "hey, that stealer log breach from ages ago now has more data".
And that's it. We'll keep backfilling data, and the entire corpus within HIBP is now cleaner and more succinct. And we'll definitely clean up all the UX and website copy as part of our impending rebrand to ensure everything is a lot clearer in the future.
I'll leave you with a bit of levity related to subscription costs and value. As I recently lamented, resellers can be a nightmare to deal with, and we're seriously considering banning them altogether. But occasionally, they inadvertently share more than they should, and we get an insight into how the outside world views the service. Like a recent case where a reseller accidentally sent us the invoice they'd intended to send the customer who wanted to purchase from us, complete with a 131% price markup 😲 It was an annual Pwned 4 subscription that's meant to be $1,370, and simply to buy this on that customer's behalf and then hand them over to us, the reseller was charging $3,165. They can do this because we make the service dirt cheap. How do we know it's dirt cheap? Because another reseller inadvertently sent us this internal communication today:
FWIW, we do have credit cards in Australia, and they work just the same as everywhere else. I still vehemently dislike resellers, but at least our customers are getting a good deal, especially when they buy direct 😊
Processing data breaches (especially big ones), can be extremely laborious. And, of course, everyone commenting on them is an expert, so there's a heap of opinions out there. And so it was with the latest stealer logs, a corpus of data that took the better part of a month to process. And then I made things confusing in various ways which led to both Disqus comment and ticket hell. But hey, it's finally out and now it's back to normal breach processing for the foreseeable future 🙂
I like to start long blog posts with a tl;dr, so here it is:
We've ingested a corpus of 1.5TB worth of stealer logs known as "ALIEN TXTBASE" into Have I Been Pwned. They contain 23 billion rows with 493 million unique website and email address pairs, affecting 284M unique email addresses. We've also added 244M passwords we've never seen before to Pwned Passwords and updated the counts against another 199M that were already in there. Finally, we now have a way for domain owners to query their entire domain for stealer logs and for website operators to identify customers who have had their email addresses snared when entering them into the site. (Note: stealer logs are still freely and easily searchable by individuals, scroll to the bottom for a walkthrough.)
This work has been a month-long saga that began hot off the heels of processing the last massive stash of stealer logs in the middle of Jan. That was the first time we'd ever added more context to stealer logs by way of making the websites email addresses had been logged against searchable. To save me repeating it all here, if you're unfamiliar with stealer logs as a concept and what we've previously done with HIBP, start there.
Up to speed? Good, let's talk about ALIEN TXTBASE.
Origin Story
Last month after loading the aforementioned corpus of data, someone in a government agency reached out and pointed me in the direction of more data by way of two files totalling just over 5GB. Their file names respectively contained the numbers "703" and "704", the word "Alien" and the following text at the beginning of each file:
Pulling the threads, it turned out the Telegram channel referred to contained 744 files of which my contact had come across just the two. The data I'm writing about today is that full corpus, published to Telegram as individual files:
The file in the image above contained over 36 million rows of data consisting of website URLs and the email addresses and passwords entered into them. But the file is just a sample - a teaser - with more data available via the subscription options offered in the message. And that's the monetisation route: provide existing data for free, then offer a subscription to feed newly obtained logs to consuming criminals with a desire to exploit the victims again. Again? The stealer logs are obtained in the first place by exploiting the victim's machine, for example:
How do people end up in stealer logs? By doing dumb stuff like this: “Around October I downloaded a pirated version of Adobe AE and after that a trojan got into my pc” pic.twitter.com/igEzOayCu6
So now this guy has malware running on his PC which is siphoning up all his credentials as they're entered into websites. It's those credentials that are then sold in the stealer logs and later used to access the victim's accounts, which is the second exploitation. Pirating software is just one way victims become infected; have a read of this recent case study from our Australian Signals Directorate:
When working from home, Alice remotely accesses the corporate network of her organisation using her personal laptop. Alice downloaded, onto her personal laptop, a version of Notepad++ from a website she believed to be legitimate. An info stealer was disguised as the installer for the Notepad++ software.
When Alice attempted to install the software, the info stealer activated and began harvesting user credentials from her laptop. This included her work username and password, which she had saved in her web browser’s saved logins feature. The info stealer then sent those user credentials to a remote command-and-control server controlled by a cybercriminal group.
Eventually, data like Alice's ends up in places like this Telegram channel and from there, it enables further crimes. From the same ASD article:
Stolen valid user credentials are highly valuable to cybercriminals, because they expedite the initial access to corporate networks and enterprise systems.
So, that's where the data has come from. As I said earlier, ALIEN TXTBASE is by no means the only Telegram channel out there, but it is definitely a major distribution channel.
Verification
When there's a breach of a discrete website, verification of the incident is usually pretty trivial. For example, if Netflix suffered a breach (and I have no indication they have, this is just an example), I can go to their website, head to the password reset field, enter a made-up email address and see a response like this:
On the other hand, an address that does exist on the service usually returns a message to the effect of "we've sent you a password reset email". This is called an "enumeration vector" in that it enables you to enumerate through a list of email addresses and find out which ones have an account on the site.
But stealer logs don't come from a single source like Netflix, instead they contain the credentials for a whole range of different sites visited by people infected with malware. However, I can still take lines from the stealer logs that were captured against Netflix and test the email addresses. (Note: I never test if the password is valid, that would be a privacy violation that constitutes unauthorised access and besides, as you'll read next, there's simply no need to.)
Initially, I actually ran into a bit of a roadblock when testing this:
I found this over and over again so, I went back and checked the source data and inspected this poor victim's record:
Their Netflix credentials were snared when they were entered into the website with a path of "/ph-en/login", implying they're likely Filipino. Let's try VPN'ing into Manilla:
And suddenly, a password reset gives me exactly what I need:
That's a little tangent from stealer logs, but Netflix obviously applies some geo-fencing logic to certain features. This actually worked out better than expected verification-wise because not only was I able to confirm the presence of the email address on their service, but that the stealer log entry placing them in the Philippines was also geographically correct. It was reproducible too: when I saw "something went wrong", but the path began with "mx", I VPN'd into Mexico City and Netflix happily confirmed the reset email was sent. Another path had "ve", so it was off to Caracas and the Venezuelan victim's account was confirmed. You get the idea. So, strong signal on confirmation of account existence via password reset, now let's also try something more personal.
I emailed a handful of HIBP subscribers and asked for their support verifying a breach. I got a fast, willing response from one guy and sent over more than 1,100 rows of data against his address 😲 It's fascinating what you can tell about a person from stealer log data: He's obviously German based on the presence of websites with a .de address, and he uses all the usual stuff most of us do (Amazon, eBay, LinkedIn, Airbnb). But it's the less common ones that make for more interesting reading: He drives a Mercedes because he's been logging into an address there for owners, and it also appears he likes whisky given his account at an online specialist. He's a Firefox user, as he's logged in there too, and he seems to be a techie as he's logged into Seagate and a site selling some very specialised electrical testing equipment. But is it legit?
Imagine the heart-in-mouth moment the poor guy must have had seeing his digital life laid out in front of him like that and knowing criminals have this data. It'd be a hell of a shock.
Having said all that, whilst I'm confident there's a large volume of legitimate data in this corpus, it's also possible there will be junk. Fabricated email addresses, websites that were never used, etc. I hope folks who experience this can appreciate how hard it is for us to discern "legitimate" stealer logs from those that were made up. We've done as much parsing and validation as possible, but we have no way of knowing if someone@yourdomain.com is an email address that actually exists or if it does, if they ever actually used Netflix or Spotify or whatever. They're just strings of data, and all we can do is report them as found.
Searching Entries Against Your Website with a Pwned 5 Subscription
When we published the stealer logs last month, I kept caveating everything with "experimental". Even the first word of the blog post title was "experimenting", simply because we didn't know how this would be received. Would people welcome the additional data we were making available? Or find it unactionable noise? It turns out it was entirely the former, and I didn't see a single negative comment or, as it often has been in the past with stealer logs or malware breaches, angry victims demanding I send their entire row(s) of data. And I guess that makes sense given what we made available so, starting today, we're making even more available!
First, a bit of nomenclature. Consider the following stealer log entry:
There are four parts to this entry that are relevant to the HIBP services I'm going to write about here:
Email Address
Website Domain
Email Alias
Email Domain
https://
www.netflix.com
/en-ph/login
john
@
example.com
P@ssw0rd
Last month, we added functionality to the UI such that after verifying your email address you could see a collection of website domains. In the example above, this meant that John could use the free notification service to verify control of his email address after which he'd see www.netflix.com listed. (Note: we're presently totally redesigning this as part of our UX rebuild and it'll be much smoother in the very near future.) Likewise, we introduced an API to do exactly the same thing, but only after verifying control of the email domain. So, in the case above, the owner of example.com would be able to query john@example.com and get back www.netflix.com (along with any other website domains poor John had been using).
Today, we're introducing two new APIs, and they're big ones:
Query stealer logs by email domain
Query stealer logs by website domain
The first one is akin to our existing domain search feature so in the example above, the owner of the domain could query the stealer logs for example.com and get back each email address alias and the website domains they appear against. Here's what that output looks like:
The previous model only allowed querying by email address, so you could end up with an organisation needing to iterate through thousands of individual API requests. This model means that can now be done in a single request, which will make life much easier for larger organisations assessing the exposure of their workforce.
The second new API is designed for website operators who want to identify customers who've had their credentials snared at login. So, in our demo row above, Netflix could query www.netflix.com (after verifying control of the domain, of course) and retrieve a list of their customers from the stealer logs:
[
"john@example.com",
"jane@yahoo.com"
]
Both these new APIs are orientated towards larger organisations and can return vast volumes of data. When considering how to price this service, the simplest, most commensurate model we arrived at was to use a pricing tier we already had: Pwned 5:
Whilst we'd previously only ever listed tiers 1 through 4, we'd always had higher tiers sitting there in the background for organisations needing higher rate limits. Surfacing this subscription and adding the ability to query stealer logs via these two new APIs makes it easy for new and existing subscribers alike to unlock the feature. And if you are an existing subscriber, the price is simply adjusted pro rata at Stripe's end such that your existing balance is carried forward. Per the above image, this subscription is available either monthly or annually so if you just want to see what's in the current corpus of data and keep the cost down, take it for a month then cancel it. (Note: the Pwned 5 subscription is also now required for the API to search by email address we launched last month, but the web UI that uses the notification service to show stealer log results by email is absolutely still free and will remain that way.)
Another small addition since last month is that we've added an "IsStealerLog" flag on the breach model. This means that anyone programmatically dealing with data in HIBP can now easily elect to handle stealer logs differently than other breaches. For example, a new breach with this flag set to "true" might then trigger a query of the new API endpoints to search by domain so that an organisation can update their records with any new stealer log entries.
Anyone searching by email domain already knows the scope of addresses on their domain as it's reported on their dashboard. Plus, when email notifications are sent on breach load it tells you exactly how many new addresses from your domains are in the breach. Starting today, we've also added a column to explain how many email addresses appear against your website domain:
In other words, 3 people have had their email address grabbed by an info stealer when logging on to hibp-integration-tests.com, and the new API will return all of those addresses. It is only API-based for the moment, we'll consider if a UI makes sense as part of the rebranded site launch, it may not becuase of the potentially huge volumes of data.
Just one last thing: for the two new APIs that query by domain, we've set a rate limit which is entirely independent of the rate limit on, say, breached account searches. Whilst a Pwned 5 subscription would allow 1,000 requests to that API every minute, it's significantly more restricted when hitting those two new stealer log APIs. We haven't published a number as I expect we'll tweak it a bit based on usage, but it's more than enough for any normal use of the service whilst ensuring we don't get completely overwhelmed by high-overhead searches. The stealer log API that queries by email address inherits the 1,000 RPM rate limit of the Pwned 5 subscription.
We've Added 244M New Passwords to Pwned Passwords
One of the coolest most awesome best things we've ever done with HIBP is to create a massive repository of passwords that's all open source (both code and data) and can be queried anonymously without disclosing the searched password. Pwned Passwords is amazing, and it has gained huge traction:
There it is - we’ve now passed 10,000,000,000 requests to Pwned Password in 30 days 😮 This is made possible with @Cloudflare’s support, massively edge caching the data to make it super fast and highly available for everyone. pic.twitter.com/kw3C9gsHmB
10 billion times a month, our API helps a service somewhere assist one of their customers with making a good password choice. And that's just the API - we have no idea the full scope of the service as having the data open source means people can just download the entire corpus and run it themselves.
Per the opening para, we now have an additional 244 million previously unseen passwords in this corpus. And, as always, they make for some fun reading:
tender-kangaroo
swimmingkangaroo59
gentlekangaroo
CaptainKangaroo340
And, uh, some kangaroos doing other stuff I can't really repeat here. Those passwords are at the final stages of loading and should flush through cache to Cloudflare's hundreds of edge nodes in the next few hours. That's another quarter of a billion that join the list of previously breached ones, whilst 199 million we'd already seen before have had their prevalence counts upped.
HIBP in Practice
It's amazing to see where my little pet project with the stupid name has gone, and nobody is more surprised than me when it pops up in cool places. Looking around for some stealer log references while writing this blog post, I came across this one:
This was already in place when you created a new account or updated your password. But now it's also verified on every login against the live HIBP database. Hats off to the tremendous service HIBP provides to the internet 🙏 https://t.co/Z61AgDaL2t
That's awesome! That's exactly the sort of use case that speaks to the motto of "do good things with breach data after bad things happen". By adding this latest trove of data, the folks using Basecamp will immediately benefit simply by virtue of the service being plugged into our API. And sidenote: David has done some amazing stuff in the past so I was especially excited to see this shout-out 😊
This one is a similar story, albeit using Pwned Passwords:
Their service is phenomenal! We also inform users in our product if they set/change their password to a known password that has been hacked. Admins have the option to not allow for users to use these passwords, if they wish. pic.twitter.com/bvLfYm9xzH
Inevitably, those requests form a slice of the 10 billion monthly we see that are now able to identify a quarter of a billion more compromised ones and hopefully, keep them out of harm's way.
For many organisations, the data we're making available via stealer logs is the missing piece of the puzzle that explains patterns that were previously unexplainable:
Gotta say I’m pretty happy with what we did with stealer logs last week, think we’re gonna need to do more of this 😎 pic.twitter.com/4rMaMmL8LU
I've had so many emails to that effect after loading various stealer logs over the years. The constant theme I love hearing is how it benefits the good guys and, well, not so much the bad guys:
The introduction of these new APIs today will finally help many organisations identify the source of malicious activity and even more importantly, get ahead of it and block it before it does damange. Whilst there won't be any set cadence to the addition of more stealer logs (obviously, we can't really predict when this stuff will emerge), I have no doubt we'll continue to add a lot more data yet.
Techie Bits
Processing this data has been non-trivial to say the least, so I thought I'd give you a bit of an overview of how I ultimately approached it. In essence, we're talking about transforming a very large amount of highly redundant text-based data and ultimately ending up with a distinct list of email addresses against website domains. Here's the high-level process:
Start with 744 files of logs totalling 1.5TB and containing 23 billion rows
In a .NET console app, process each file in point 1 and extract the domain and email address from valid lines (domain, email and password, all colon delimited) to produce 744 files totalling 390GB (9.7B rows)
In another .NET console app, consolidate all 744 files from the previous point into a single file with a distinct set of website domain and email address pairs (789M rows)
Take the file from the previous point, and in another .NET console app, extract all the unique domains (18M rows)
Use SQL BCP to upload the files from the two previous points to SQL Azure
Insert any new domains that don't already exist in HIBP (these are held in a dedicated table) via a TSQL statement (6.7M rows)
Upload the 284M unique email addresses like with a typical data breach (69% of them were already in HIBP)
Join the distinct list of domains and email addresses to the data uploaded in the previous point and insert email and domain pairs into the stealer log table, but only if they haven't been seen before via another TSQL statement (493M rows)
Wait - if I took distinct website and email pairs in step 4 and got 789M rows then in step 9 it only inserted 493M, what happened? There were 220M rows already in HIBP from last month which will account for some of the gap (existing records aren't reinserted), and there was also some additional validation in SQL Server courtesy of code we only have in that environment. The remaining gap is explained by the .NET code not ignoring case on distinct, so in other words, I dumped and uploaded way more data than I had to and made SQL Sever do extra work 🤦♂️
Go live and get 🍺
It was actually much, much harder than this due to the trials and errors of attempting to distil what starts out as tens of billions of rows down into hundreds of millions of unique examples. I was initially doing a lot more of the processing in SQL Azure because hey, it's just cloud and you can turn up the performance as much as you want, right?! After running 80 cores of Azure SQL Hyperscale for days on end ($ouch$) and seeing no end in sight to the execution of queries intended to take distinct values across billions of rows, I fell back to local processing with .NET. You could, of course, use all sorts of other technologies of choice here, but it turned out that local processing with some hand-rolled code was way more efficient than delegating the task to a cloud-based RDBMS. The space used by the database tells a big part of the story:
As I've said many times before, the cloud, my friends, is not always the answer. Do as much processing as possible locally on sunk-cost hardware unless there's a compelling reason not to.
I've detailed all this here in part because that's what I've always done with this project over the last 11 and a bit years, but also to illustrate how much time, effort, and money is burned processing data like this. It's very non-trivial, especially not when everything has to ultimately go into an increasingly large system with loads of external dependencies and be fast, reliable and cost-effective.
Conclusion
From 23 billion raw rows down to a much more manageable and discrete set of data, the latest stealer logs are now searchable via all the ways you've done for years, plus those two new domain-based stealer log APIs. We've called this breach "ALIEN TXTBASE Stealer Logs", let's see what positive outcomes we can all now produce with the data.
Edit 1: Let me re-emphasise an important point from the blog post I think got a bit buried: The web UI that uses the notification service to show stealer log results by email is absolutely still free and will remain that way. If you’ve got an email address in this breach and you want to see the stealer log domains against it, do this:
Fill in your email address and send yourself the verification email
Click the link emailed to you and scroll to the bottom of the page where you'll find a message similar to this:
We need to make this clearer as it's obviously confusing. Thanks for everyone's feedback, we're working on it.
Edit 2:I've just published a (much shorter!) blog post that addresses a common theme in the comments regarding email addresses that only appear against a single website domain, being the same domain as the email address itself is on. Check that out if that's you.
Wait - it's Tuesday already?! When you listen to this week's (ok, last week's) video, you'll probably get the sense I was a bit overloaded. Yeah, so that didn't stop, and the stealer log processing and new feature building just absolutely swamped me. Plus, I spent from then until now in Sydney at various meetings and events which was great, but didn't do a lot for my productivity. Be that as it may, we're now less than 12 hours off launching this all so, in the interests of not having me stay up all night putting the finishing touches on it, let me drop here and come back in a few days to talk about how it's all been received 🤞
Login
