Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10416

Getting error "message" : "The KMS key does not contain a location." while using withKmsKey() option at runtime in dataflow job

Details

    Description

       
      I have dataflow job that is written using apache beam.Here I am loading the data from one table to another table the mode of table is write truncate and the table contains sensitive data so table configured with kms key. I am following this code and trying to fetch the kms key associated with table and setting this key at the time of (BigQueryIO.write) while writing data into table.

      https://github.com/apache/beam/blob/49403c00499f7f87c2bf9a4c63dec8ebd68d640d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java#L91-L100

       

       

      {{finalOutput.apply("Write success rows to Sensitive BigQuery",
      BigQueryIO.writeTableRows().withoutValidation()
      .to(options.getTargetTable())
      .withKmsKey(NestedValueProvider.of(options.getTargetTable(),new FetchingKMSKey(options.getProject())).toString()).withoutValidation()
      .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
      .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));}}

       

       

      {{public class FetchingKMSKey implements SerializableFunction<String, String>

      {

      private static final BigqueryClient BQ_CLIENT = new BigqueryClient("dataMainPipeline");

      public static final Logger LOG = LoggerFactory.getLogger(FetchingKMSKey.class);
      Table table ;
      String tableName;
      String project;
      public FetchingKMSKey(String project)

      { this.project=project; }

      public String apply(String tableName)
      {
      String[] name= tableName.split(Pattern.quote("."));

      try

      { table = BQ_CLIENT.getTableResource(project,name[0],name[1]); }

      catch (IOException e)

      { LOG.error(String.format("exception occured: %s", e.getMessage())); }

      catch (InterruptedException e)

      { LOG.error(String.format("exception occured: %s", e.getMessage())); }

      String kmsKey =table.getEncryptionConfiguration().getKmsKeyName();
      return kmsKey;

      }

      }}}

      {{}}

       

      After creating a template when I am running the job I am getting error.

       

      {{"location" : "US",

      "errors" : [

      { "message" : "The KMS key does not contain a location.", "reason" : "invalid" }

      ],}}

       

      I am completely stuck. anyone could help?

      Attachments

        Activity

          People

            Unassigned Unassigned
            user-a27 Arzoo Khan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: