Is there any way to detect strings like putjbtghguhjjjanika?

by

You could build a model of character to character transitions from a bunch of text in English. So for example, you find out how common it is for there to be a ‘h’ after a ‘t’ (pretty common). In English, you expect that after a ‘q’, you’ll get a ‘u’. If you get a ‘q’ followed by something other than a ‘u’, this will happen with very low probability, and hence it should be pretty alarming. Normalize the counts in your tables so that you have a probability. Then for a query, walk through the matrix and compute the product of the transitions you take. Then normalize by the length of the query. When the number is low, you likely have a gibberish query (or something in a different language).

If you have a bunch of query logs, you might first make a model of general English text, and then heavily weight your own queries in that model training phase.

For background, read about Markov Chains.

Edit, I implemented this here in Python:

https://github.com/rrenaud/Gibberish-Detector

and buggedcom rewrote it in PHP:

https://github.com/buggedcom/Gibberish-Detector-PHP

my name is rob and i like to hack True
is this thing working? True
i hope so True
t2 chhsdfitoixcv False
ytjkacvzw False
yutthasxcvqer False
seems okay True
yay! True

More Related Contents:

  1. How to generate all permutations of a string in PHP?
  2. String with array structure to Array [duplicate]
  3. finding common prefix of array of strings
  4. Count array by parse value of array
  5. i need a PHP code to find longest contiguous sequence of characters in the string
  6. How can I get a hex dump of a string in PHP?
  7. Remove new lines from string and replace with one empty space
  8. PHP, get file name without file extension
  9. URL Friendly Username in PHP?
  10. PHP: Split string [duplicate]
  11. Mixing a PHP variable with a string literal
  12. PHP equivalent of .NET/Java’s toString()
  13. How to mimic Stack Overflow Auto-Link Behavior
  14. how to replace special characters with the ones they’re based on in PHP?
  15. How can I truncate a string to the first 20 words in PHP?
  16. Convert flat array to a delimited string to be saved in the database
  17. Split string by new line characters
  18. How to get email address from a long string
  19. String contains any items in an array (case insensitive)
  20. Algorithm to get all possible string combinations from array up to certain length
  21. How to cast array elements to strings in PHP?
  22. How to reverse a Unicode string
  23. remove duplicate from string in PHP
  24. How to Convert Boolean to String
  25. Delete first 3 characters and last 3 characters from String PHP
  26. How can I get the last 7 characters of a PHP string?
  27. Convert a String to Variable
  28. How to convert all characters to their html entity equivalent using PHP
  29. Remove first 4 characters of a string with PHP
  30. PHP Find all occurrences of a substring in a string

Leave a Comment