Facebook's WhatsApp sued NSO in October after finding evidence that the hacking company had abused a flaw in the popular chat program to remotely hijack hundreds of smartphones.
After NSO failed to appear in court, a California clerk of court filed a default notice against NSO last week.
But NSO says it had not been complied with in accordance with international law known as the Hague Convention.
Facebook "lied to the court in its request for non-compliance by declaring that the defendants had been notified under the Hague Convention, when, in fact, the Israeli government had told the plaintiffs two days before the service under the Hague Convention it was not complete, and the service application had to be resubmitted, "documents filed by NSO in the Northern District of California court said this week.
NSO asked the court to impose penalties of nearly $ 17,000 to cover the attorneys' fees and costs.
Facebook also filed documents this week requesting that the court nullify the breach entry so the case can proceed on merit.
Facebook said it "diligently sought,quot; to serve the OSN in accordance with the Hague Convention, but the Israel Central Authority has yet to issue a formal certificate of service.
"The court should reject the defendants' request to further delay this case," Facebook said in the documents.
The Facebook lawsuit, along with the alert messages WhatsApp sent to hundreds of users suspected of being hacked, prompted revelations about NSO's surveillance work around the world.
The case has been closely watched, both from the high-tech surveillance angle and also because it was unusual for a major service provider to sue a hacking company on behalf of its users.