LogFAQs > #956893513

some clarification has been released.
there are actually 2 separate filters, for different purposes.

an AI that detects adult content you send or receive through the messaging app or search for in siri:
this is only active for minors that are part of a family group/account, and only when an adult in the group enables it.
this will produce a warning that is not shared with authorities or apple, if the minor is under 12 it'll be sent to adults in the group, if the minor is older it'll only be shown on the minor's device.
this AI "learns" and will analyze the actual content, and flags any mature content.
I think the siri part is always on for everyone, but only produces a warning for the user.("this search query may result in offensive results, are you sure you want to perform this search?")

and a fingerprint check for icloud photo library uploads:
this most likely uses a technology similar to tineye.com or soundcloud, it reduces a photo's core details to a series of numbers that are unique to that photo and any minor edits of it(mirrors, resizes, crops, color shifts, etc.)
this fingerprint is compared to a list of fingerprints of confirmed "bad" photos.
if a photo matches a fingerprint an encrypted copy and an unencrypted thumbnail are put in an encrypted vault in your icloud, this encrypted vault cannot be decrypted by anyone until it "overflows"(when a certain amount of flagged images are put into it)
when the "vault" is full it gets sent to a team of Apple employees who can then decrypt it and check the thumbnails to see wether there are any false positives, they never get access to the full sized images and apple estimates that there's a chance of 1 in 1 trillion every year that there will be enough false positives to fill up such a vault.
only after that the decrypted vault will be sent for final verification to the only organization in the US allowed to process these "bad" images, and from there the authorities will be notified.
Apple promises that they'll refuse any request or order from governments to add other types of "unwanted images" to the database for this filter.