PropScreen's LLM Implementation
Last updated
Last updated
An alternative architecture has been proposed where instead of a series of checks, PropScreen would instead itself be a language model that has been given the context of the organizational sensitive information and has been tasked with reading the responses of the organization's LLM and determining whether or not the responses contain organizational sensitive information. PropScreen's LLM would not directly interact with the LLM user, but make a determination on the existence of organization sensitive information a the Company LLM's response. The determination would then dictate the action that PropScreen takes in the first use case.
This approach would subject the organization's LLM responses to a single check instead of a series of three checks. While still in speculation, it is likely that the single check that the PropScreen model would perform would be faster that the sum of the NER model, regex, and two checks against databases that PropScreen currently utilizes.
A language model scanning for organization sensitive information being disclosed inside of responses could be more dynamic in its detection of sensitive information. The primary reason being is that a model would be able to detect organizational data that has been misformatted or obfuscated in some way. For example, if client IDs are considered sensitive information and one such ID is "ZB-743223", a hashed check would fail against a response that contains "ZB -743 223" - a model has the possibility of identifying this correctly as the client ID "ZB-743223" and blocking it.
This approach would require PropScreen to have access to the organization's sensitive information in the clear text. While this is not prohibitive for the solution, the PropScreen's model knowing organizational sensitive information should be considered when determining the tolerable levels of risk for this type of implementation.