공식 환경의 데이터 마스크를 제작한 후 자동으로 무대 환경과 동기화하는 메커니즘 [AWS CDK/SDK] 3~SDK 편 후반~

177360 단어 AWS sdk DB CDK tech

저번

섹션 1: 공식 환경의 데이터 마스크를 자동으로 스테이지 환경과 동기화하는 메커니즘 [AWS CDK/SDK] 1~CDK 구축 편~
섹션 2: 공식 환경을 가리는 데이터를 제작하여 무대 환경과 자동으로 동기화하는 구조[AWS CDK/SDK] 2~SDK 편 전반~

개시하다

나는 공기 벽장 회사에서 엔지니어로 일하는 삼호이다.
이번 이야기는 공식 환경의 데이터를 마스크하고 무대 환경과 자동으로 동기화하는 메커니즘을 구축하는 것으로 지난번의 계속이다.
1부에서는 이번 목적과 요건에 따라 AWS CDK를 통해 전체적인 구성을 구축했다
2부에서는 실제 램바다 처리의 전반부를 AWS SDK for JavaScript로 썼다.
마지막으로 세 번째 부분의 이 글에서 람바다 처리의 후반부를 썼다.

3. 테이블별 Masking 처리 수행

Point

ORM

ORM 도입도 논의됐지만, 마스킹 자체가 복잡한 처리가 아니기 때문에 오리지널 MySQL/PostgreSQL로 처리했다.

Masking의 임의 값

테이블에 따라 Masking 이후의 임의 값을 입력하려는 경우가 있어 복잡해지는 테이블도 있습니다.

Chunk

처리가 너무 무거우면 Database의 연결이 끊기기 때문에 Chunk 처리를 하는 표도 있다.당연한 일이지만 실행이 길어질 수 있으니 하지 않는 게 좋다.

NULL 열

Masking에서는 원래 NULL이던 열이 NULL을 유지합니다.(공식 데이터에 접근하기 위해)

query vs execute

connection의query 방법은 SQL 주입을 할 수 있지만,execute 방법이라면 처리가 중도에 끝날 수 있기 때문에query를 사용합니다.

postgres role

조회를 통해 오래된 제본 DB에서 덤프(pg dummpall)를 실행하려고 했지만 권한상 실행할 수 없기 때문에 Spread Sheet에서 연결 정보를 얻고Create User를 실행합니다.연구한 프로그램 라이브러리pgdump-aws-lambda를 사용합니다.

ALL PRIVILAGES라면 CREATERROLE 권한이 없습니다.

presentation 레이어

presentation/handler/anonymization-task.ts

import { Context } from 'aws-lambda';
import { AnonymizeDatabaseUsecase } from '../../usecase/anonymize-database';
import { send }  from '../../infrastructure/slack';

exports.handler = async (event: any, context: Context) => {
  const anonymizeDatabaseUsecase = new AnonymizeDatabaseUsecase();

  try {
    await anonymizeDatabaseUsecase.execute(event);
  } catch (error) {
    await send('#developers', `Failed to anonymize database for staging environment. error: ${error}`);
    console.log({ status: 500, event, error });
    return { status: 500, event, error };
  };

  console.log({ status: 200, event });
  return { status: 200, event };
};

usesease 레이어

usecase/anonymize-database.ts

import { IAnonymization } from '../@types';

export class AnonymizeDatabaseUsecase {
  async execute({ database, task }: {
    database: {
      name: string,
      instanceId: string,
      host: string,
      username: string,
      password: string,
    },
    task: string,
  }) {
    const { anonymize }: IAnonymization = await import(
      `../domain/repository/database/anonymization/${database.name}/${task}`
    );
    const databaseConnectInfo = {
      name: database.name,
      instanceId: database.instanceId,
      host: database.host,
      username: database.username,
      password: database.password,
    };

    return await anonymize(databaseConnectInfo);
  }
}

repository 레이어

Macking 예

다음 파일은 Masking 처리, 각 Database 및 각 테이블의 이미지입니다.
모든 테이블의 Macking 방법에는 각양각색의 예가 있다.
domain/repository/database/anonymization/database1/users.ts

import { DatabaseRepository } from '../../../database';
import { QueryRepository } from '../query';
import { IDatabase } from '../../../../../@types';

export const anonymize = async (database: IDatabase) => {
  try {
    const databaseRepository: DatabaseRepository = new DatabaseRepository();
    const queryRepository: QueryRepository = new QueryRepository();
    const connection = await databaseRepository.connectionDatabase(database);
    if (!connection) {
      throw new Error('Databaseへの接続に失敗しました');
    }

    const tableName = 'users';

    const updateSql = `
      UPDATE ${tableName}
      SET ${tableName}.updated_at = NOW(),
          ${tableName}.${queryRepository.getLoginEmail(tableName, 'email')},
          ${tableName}.${queryRepository.getPassword(
            'password',
          )}
      ;
    `;

    await connection.query(updateSql);
    await databaseRepository.disConnectionDatabase(connection, database.name);
    return;
  } catch (error) {
    throw new Error(error);
  }
};

domain/repository/database/anonymization/database1/user_information.ts

import { DatabaseRepository } from '../../../database';
import { AnonymizationRepository } from '..';
import { QueryRepository } from '../query';
import { IDatabase } from '../../../../../@types';

export const anonymize = async (database: IDatabase) => {
  try {
    const anonymizationRepository: AnonymizationRepository = new AnonymizationRepository();
    const queryRepository: QueryRepository = new QueryRepository();
    const databaseRepository: DatabaseRepository = new DatabaseRepository();

    const connection = await databaseRepository.connectionDatabase(database);
    if (!connection) {
      throw new Error('Databaseへの接続に失敗しました');
    }

    const tableName = 'user_information';
    const findAllSql = `
      SELECT id, nick_name, fist_name, last_name, fist_name_kana, last_name_kana, address, tel, profile_picture, face_picture
      FROM ${tableName}
      ;
    `;

    const [rows] = await connection.query(findAllSql);

    // memo: 処理が重すぎて失敗することがあったので、レコード数が多かった場合はChunkする
    const isChunkPromise = anonymizationRepository.getIsChunkPromise(rows);

    // memo: 一定数でChunkして処理を実行
    const chunkPromiseCount = anonymizationRepository.getChunkPromiseCount();

    const updateGroup: any = {};
    let updateSqlList = [];

    // memo: ランダムとなる値のすべての組み合わせを抽出
    for (const userInformation of rows) {
      let lastName = anonymizationRepository.getFakeUserLastName(
        userInformation.last_name,
      );
      let firstName = anonymizationRepository.getFakeUserFirstName(
        userInformation.first_name,
      );
      let facePicture = anonymizationRepository.getDummyFacePicture(
        userInformation.face_picture,
      );
      let profilePicture = anonymizationRepository.getDummySystemicPicture(
        userInformation.profile_picture,
      );

      updateGroup[lastName || 'null'] = updateGroup[lastName || 'null'] || {};
      updateGroup[lastName || 'null'][firstName || 'null'] =
        updateGroup[lastName || 'null'][firstName || 'null'] || {};
      updateGroup[lastName || 'null'][firstName || 'null'][
        facePicture || 'null'
      ] =
        updateGroup[lastName || 'null'][firstName || 'null'][
          facePicture || 'null'
        ] || {};
      updateGroup[lastName || 'null'][firstName || 'null'][
        facePicture || 'null'
      ][profilePicture || 'null'] =
        updateGroup[lastName || 'null'][firstName || 'null'][
          facePicture || 'null'
        ][profilePicture || 'null'] || [];
      updateGroup[lastName || 'null'][firstName || 'null'][
        facePicture || 'null'
      ][profilePicture || 'null'].push(userInformation.id);
    }
    let idx = 0;
    const lastNameKeys = Object.keys(updateGroup);

    for (let i = 0, length = lastNameKeys.length; i < length; i++) {
      const lastName = lastNameKeys[i];
      const firstNameKeys = Object.keys(updateGroup[lastName]);

      for (let k = 0, length = firstNameKeys.length; k < length; k++) {
        const firstName = firstNameKeys[k];
        const facePictureKeys = Object.keys(updateGroup[lastName][firstName]);

        for (let m = 0, length = facePictureKeys.length; m < length; m++) {
          const facePicture = facePictureKeys[m];
          const profilePictureKeys = Object.keys(
            updateGroup[lastName][firstName][facePicture],
          );

          for (let j = 0, length = profilePictureKeys.length; j < length; j++) {
            const profilePicture = profilePictureKeys[j];
            const updateIds =
              updateGroup[lastName][firstName][facePicture][profilePicture];

            const updateSql = `
              UPDATE ${tableName}
              SET ${tableName}.updated_at = NOW(),
                  ${tableName}.last_name = ${lastName},
                  ${tableName}.fist_name = ${firstName},
                  ${tableName}.nick_name = ${firstName},
                  ${tableName}.face_picture = ${facePicture},
                  ${tableName}.profile_picture = ${profilePicture},
                  ${tableName}.${queryRepository.getLastNameKana(
              'last_name_kana',
            )},
                  ${tableName}.${queryRepository.getFirstNameKana(
              'fist_name_kana',
            )},
                  ${tableName}.${queryRepository.getStreet('address')},
                  ${tableName}.${queryRepository.getTel('tel')},
              WHERE ${tableName}.id IN(${updateIds.join()});
            `;

            updateSqlList.push(updateSql);
            idx++;

            if (isChunkPromise) {
              if (idx !== 0 && idx % chunkPromiseCount === 0) {
                await Promise.all(
                  updateSqlList.map((sql) => connection.query(sql)),
                );
                updateSqlList = [];
              }
            } else {
              await Promise.all(
                updateSqlList.map((sql) => connection.query(sql)),
              );
            }
          }
        }
      }
    }
    if (idx > 0) {
      await Promise.all(updateSqlList.map((sql) => connection.query(sql)));
    }

    await databaseRepository.disConnectionDatabase(connection, database.name);
    return;
  } catch (error) {
    throw new Error(error);
  }
};

Masking 사용 방법의 예

domain/repository/database/anonymization/index.ts

import {
  getFakeFirstName,
  getFakeLastName,
} from '../../../../infrastructure/faker';
import { getRandomYmd } from '../../../../utils';

export class AnonymizationRepository {
  public getChunkPromiseCount(): number {
    return 100; // memo: 固定値
  }

  public getIsChunkPromise(records: object[]): boolean {
    return records.length > 500; // memo: 固定値。1000では動かなくなるテーブルがあった経緯。
  }

  // memo: ランダムで名前を取得
  public getFakeUserLastName(lastNameData: string | null) {
    let lastName = null;
    if (lastNameData) {
      lastName = `'${getFakeLastName()}'`;
    }
    return lastName;
  }

  public getFakeUserFirstName(firstNameData: string | null) {
    let firstName = null;
    if (firstNameData) {
      firstName = `'${getFakeFirstName()}'`;
    }
    return firstName;
  }

  public getDummyUserLastNameKana(lastNameKanaData: string | null) {
    let lastNameKana = null;
    if (lastNameKanaData) {
      lastNameKana = `'ミョウジ'`;
    }
    return lastNameKana;
  }

  public getDummyUserFirstNameKana(firstNameKanaData: string | null) {
    let firstNameKana = null;
    if (firstNameKanaData) {
      firstNameKana = `'ナマエ'`;
    }
    return firstNameKana;
  }

  public getDummyBirthday(birthdayData: string | null) {
    let birthday = null;
    if (birthdayData) {
      birthday = `'${getRandomYmd('1950/01/01', '2000/12/31')}'`;
    }
    return birthday;
  }

  public getDummyStreet(streetData: string | null) {
    let street = null;
    if (streetData) {
      street = `'1-1-1'`;
    }
    return street;
  }
}

domain/repository/database/anonymization/query.ts

export class QueryRepository {
  public getDummyEmail(tableName: string, columnName: string) {
    return `
      ${columnName} =
        CASE WHEN ${columnName} IS NOT NULL THEN (
          CONCAT('test_' , CAST(${tableName}.id AS CHAR) , '@hoge.com')
        ) ELSE NULL END
    `;
  }

  public getStreet(columnName: string) {
    return ` ${columnName} = CASE WHEN ${columnName} IS NOT NULL THEN '1-1-1' ELSE NULL END `;
  }

  public getFirstName(columnName: string) {
    return ` ${columnName} = CASE WHEN ${columnName} IS NOT NULL THEN '太郎' ELSE NULL END `;
  }

  public getFirstNameKana(columnName: string) {
    return ` ${columnName} = CASE WHEN ${columnName} IS NOT NULL THEN 'ナマエ' ELSE NULL END `;
  }

  public getBirthday(columnName: string) {
    return ` ${columnName} = CASE WHEN ${columnName} IS NOT NULL THEN '2000-12-25 00:00:00' ELSE NULL END `;
  }
}

일반 처리

domain/repository/database/index.ts

import { MySqlRepository } from './mysql';
import { PostgresRepository } from './postgres';
import { DATABASE_CONFIG } from '../../../config/database';
import { IDatabase } from '../../../@types';

export class DatabaseRepository {
  public async connectionDatabase(database: IDatabase) {
    const mySqlRepository: MySqlRepository = new MySqlRepository();
    const postgresRepository: PostgresRepository = new PostgresRepository();
    let connection: any;

    switch (database.name) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DATABASE_NAME:
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DATABASE_NAME:
        connection = await mySqlRepository.createPool(database);
        break;
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DATABASE_NAME:
        connection = await postgresRepository.connection(database);
        break;
      default:
        throw new Error(
          `DatabaseName: ${database.name} は接続できないDatabaseです`,
        );
    }
    if (!connection) {
      throw new Error(
        `Database識別子: ${database.instanceId} へ接続に失敗しました`,
      );
    }
    return connection;
  }

  public async disConnectionDatabase(connection: any, databaseName: string) {
    const mySqlRepository: MySqlRepository = new MySqlRepository();
    const postgreRepository: PostgresRepository = new PostgresRepository();

    switch (databaseName) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DATABASE_NAME:
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DATABASE_NAME:
        return await mySqlRepository.disConnection(connection);
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DATABASE_NAME:
        return await postgreRepository.disConnection(connection);
      default:
        throw new Error('切断できないDatabaseです');
    }
  }
}

domain/repository/database/mysql/index.ts

import { createPool, disConnection, } from '../../../../infrastructure/mysql';
import { IDatabase } from '../../../../@types';

export class MySqlRepository {
  public async createPool(database: IDatabase) {
    return await createPool(database);
  }

  public async disConnection(connection: any) {
    return await disConnection(connection);
  }
};

domain/repository/database/postgres/index.ts

import { IDatabase } from '../../../../@types';
import { connection, disConnection } from '../../../../infrastructure/postgres';

export class PostgresRepository {
  public async connection(database: IDatabase) {
    return await connection(database);
  };

  public async disConnection(connection: any) {
    return await disConnection(connection);
  };
};

infrastrue 레이어

infrastructure/mysql.ts

import * as mysql from 'mysql2/promise';
import { IDatabase } from '../@types';

export const createPool = async (database: IDatabase) => {
  let pool!: mysql.Pool;
  pool = await mysql.createPool({
    host: database.host,
    user: database.username,
    password: database.password,
    database: database.name,
    connectionLimit: 100,
    connectTimeout: 1500000,
  });
  pool.getConnection();
  return pool;
};

export const disConnection = async (connection: mysql.Pool | mysql.Connection) => {
  return await connection.end();
};

infrastructure/postgres.ts

import pg from 'pg';
import { IDatabase } from '../@types';

export const connection = async (database: IDatabase) => {
  let pool!: pg.Pool;
  const config = {
    host: database.host,
    user: database.username,
    password: database.password,
    database: database.name,
    port: 5432,
    idleTimeoutMillis: 6000000,
  };

  pool = new pg.Pool(config);
  await pool.connect()
  return pool;
};

export const disConnection = async (connection: any) => {
  return (await connection.connect()).release();
};

4. 후처리

할 일

기존 DB의 연결 사용자 정보를 신규 DB에 추가

호스팅된 연결 사용자에 의해 연결 확인

DNS 재설치

기존 호스팅 DB 삭제

Point

MySQL Insert에서 사용자를 연결한 후FLUSH PRIVILEGES;는 연결 정보를 반영한다

mysql 연결 사용자

rdsadminrdsrepladmin: Delete 불가

prod-restore-rds4staging: 연결된 자신이 되기 때문에 Delete를 삭제하지 않습니다.

StopFunction 및 Lambda 시간 초과 메커니즘

Lambda 시간 초과 전에 stepFunction 시간 초과를 맞이하는 경우

Step Function 자체는 시간 초과 처리로 간주됩니다.단계 Function에서 Lambda 판정이 취소되었습니다.

presentation 레이어

presentation/handler/anonymized-database-task.ts

import { Context } from 'aws-lambda';
import { ConfigureDatabaseUsecase } from '../../usecase/configure-database';
import { send }  from '../../infrastructure/slack';
import { IDatabase } from '../../@types';

exports.handler = async (event: any, context: Context) => {
  const configureDatabaseUsecase = new ConfigureDatabaseUsecase();
  const databaseConnectInfo: IDatabase = {
    name: event.name,
    instanceId: event.instanceId,
    host: event.host ? event.host : event.host.Address, // memo: clusterとinstanceでプロパティの持ち方が異なる
    username: event.username,
    password: event.password,
  };

  try {
    await configureDatabaseUsecase.execute(databaseConnectInfo);
  } catch (error) {
    await send('#developers', `Failed to configure database for staging environment. error: ${error}`);
    console.log({ status: 500, event, error });
    return { status: 500, event, error };
  }

  console.log({ status: 200, event });
  return { status: 200, event };
};

usesease 레이어

import { RdsRepository } from '../domain/repository/aws/rds';
import { DatabaseRepository } from '../domain/repository/database';
import { IDatabase } from '../@types';
import { Route53Repository } from '../domain/repository/aws/route53';
import { SsmRepository } from '../domain/repository/aws/ssm';
import { CONFIG } from '../config';
import { SpreadsheetRepository } from '../domain/repository/spreadsheet';

export class ConfigureDatabaseUsecase {
  async execute(afterDBConnectionInfo: IDatabase) {
    const rdsRepository: RdsRepository = new RdsRepository();
    const route53Repository: Route53Repository = new Route53Repository();
    const databaseRepository: DatabaseRepository = new DatabaseRepository();
    const ssmRepository: SsmRepository = new SsmRepository();
    const spreadsheetRepository: SpreadsheetRepository = new SpreadsheetRepository();

    const targetDBName: string = afterDBConnectionInfo.name;
    const dbIdentifier: string = afterDBConnectionInfo.instanceId;
    const hostedZone = await route53Repository.getTargetHostedZone(
      CONFIG.HOSTED_ZONE.NAME,
    );
    const beforeHostName = await route53Repository.getHostNameFromRoute53(
      targetDBName,
      hostedZone,
    );
    const mysqlEngines = ['mysql', 'aurora-mysql'];
    const postgresqlEngines = ['postgres', 'aurora-postgresql'];

    let matchedIdentifier: RegExpMatchArray | null;
    matchedIdentifier = beforeHostName.match(/(.*)-\d{12}/);

    if (!matchedIdentifier) {
      throw new Error(
        `DatabaseName: ${afterDBConnectionInfo.name} のDB識別子が取得できませんでした`,
      );
    }

    const beforeDBDescribe = await rdsRepository.postDescribedDatabaseFromEndpoint(
      matchedIdentifier,
    );
    const {
      username,
      password,
    } = await ssmRepository.postUsernameAndPassword4Staging(targetDBName);
    const beforeDBConnectionInfo = {
      name: targetDBName,
      instanceId: matchedIdentifier[0],
      host: beforeHostName,
      username: username,
      password: password,
    };

    const afterDBConnection = await databaseRepository.connectionDatabase(
      afterDBConnectionInfo,
    );
    const beforeDBConnection = await databaseRepository.connectionDatabase(
      beforeDBConnectionInfo,
    );

    const isMysqlEngine = mysqlEngines.includes(beforeDBDescribe.Engine);
    const isPostgresqlEngine = postgresqlEngines.includes(
      beforeDBDescribe.Engine,
    );

    if (isMysqlEngine) {
      await databaseRepository.deleteConnectionUsers(
        afterDBConnection,
        beforeDBDescribe.Engine,
      );
      const connectionUsers = await databaseRepository.selectConnectionUsers(
        beforeDBConnection,
        beforeDBDescribe.Engine,
      );
      await databaseRepository.insertConnectionUsers(
        targetDBName,
        afterDBConnection,
        beforeDBDescribe.Engine,
        connectionUsers,
      );
    } else if (isPostgresqlEngine) {
      const createRoleSqls = await spreadsheetRepository.getCreateRoleSqls(
        process.env.SPREADSHEET_ID || '',
        process.env.WORKSHEET_ID || '',
      );
      await databaseRepository.insertConnectionUsers(
        targetDBName,
        afterDBConnection,
        beforeDBDescribe.Engine,
        createRoleSqls,
      );
    } else {
      throw new Error('接続ユーザーの追加に失敗しました');
    }

    // memo: Insertしたユーザーで接続できるかを確認するために旧DBと新DBとの接続を一度切る
    await databaseRepository.disConnectionDatabase(
      beforeDBConnection,
      beforeDBConnectionInfo.name,
    );
    await databaseRepository.disConnectionDatabase(
      afterDBConnection,
      afterDBConnectionInfo.name,
    );

    // memo: 新しいDatabaseにステージング用のユーザーで接続できるか確認
    await databaseRepository.checkDatabaseConnection(
      afterDBConnectionInfo,
      beforeDBDescribe.Engine,
      username,
      password,
    );

    // memo: DNSの設定（該当のstg環境のCNAMEを新しくrestoreされたDBに付け替える）。
    const dbDescribe = await rdsRepository.postDescribedDatabaseFromDBName(
      targetDBName,
      dbIdentifier,
    );
    const endpoint: string = dbDescribe.Endpoint.Address
      ? dbDescribe.Endpoint.Address
      : dbDescribe.Endpoint;
    if (!endpoint) {
      throw new Error('RDSのEndpointが取得できませんでした');
    }
    await route53Repository.route53ChangeCurrentDBToRestoredDB(
      targetDBName,
      endpoint,
      hostedZone.Id,
    );

    // memo: 旧DBの削除
    await rdsRepository.deleteDatabase(matchedIdentifier);
    return;
  }
}

repository 레이어

domain/repository/aws/rds.ts

import {
  describeDBClusters,
  describeDBInstances,
  deleteDBCluster,
  deleteDBInstance,
} from '../../../infrastructure/aws/rds';
import { DATABASE_CONFIG } from '../../../config/database';

export class RdsRepository {
  public async postDescribedDatabaseFromDBName(
    dbName: string,
    identifier: string,
  ) {
    let describeDatabase: any;
    switch (dbName) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DATABASE_NAME:
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DATABASE_NAME:
        const describeDatabaseCluster = await describeDBClusters(identifier);
        describeDatabase = describeDatabaseCluster?.DBClusters?.[0];
        break;
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DATABASE_NAME:
        const describeDatabaseInstance = await describeDBInstances(identifier);
        describeDatabase = describeDatabaseInstance?.DBInstances?.[0];
        break;
      default:
        throw new Error(
          '想定外のDatabaseDescribeを取得しようとしたため処理を終了します',
        );
    }
    return describeDatabase;
  }

  public async postDescribedDatabaseFromEndpoint(
    matchedIdentifier: RegExpMatchArray,
  ) {
    const targetIdentifier = matchedIdentifier[0];
    const commonIdentifier = matchedIdentifier[1];

    let describeDatabase: any;
    switch (commonIdentifier) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DB_IDENTIFIER.STAGING:
        const describeDatabaseCluster = await describeDBClusters(
          targetIdentifier,
        );
        describeDatabase = describeDatabaseCluster?.DBClusters?.[0];
        break;
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DB_IDENTIFIER.STAGING:
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DB_IDENTIFIER.STAGING:
        const describeDatabaseInstance = await describeDBInstances(
          targetIdentifier,
        );
        describeDatabase = describeDatabaseInstance?.DBInstances?.[0];
        break;
      default:
        throw new Error('想定外の識別子のためDB情報取得ができません');
    }
    if (!describeDatabase) {
      throw new Error('DB情報の取得に失敗しました');
    }
    return describeDatabase;
  }

  public async deleteDatabase(matchedIdentifier: RegExpMatchArray) {
    const targetIdentifier = matchedIdentifier[0];
    const commonIdentifier = matchedIdentifier[1];

    switch (commonIdentifier) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DB_IDENTIFIER.STAGING:
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DB_IDENTIFIER.STAGING:
        return await deleteDBCluster(targetIdentifier);
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DB_IDENTIFIER.STAGING:
        return await deleteDBInstance(targetIdentifier);
      default:
        throw new Error('対象外のDatabaseは削除できません');
    }
  }
}

domain/repository/database/index.ts

import { DATABASE_CONFIG } from '../../../config/database';
import { IDatabase } from '../../../@types';

export class DatabaseRepository {
  public async deleteConnectionUsers(
    connection: any,
    engine: string,
    users?: any[],
  ) {
    switch (engine) {
      case 'mysql':
      case 'aurora-mysql':
        const deleteUsers4MySql = `DELETE FROM mysql.user WHERE User NOT IN ('rdsadmin', 'rdsrepladmin', 'prod-restore-rds4staging') ;`;
        return await connection.execute(deleteUsers4MySql);
      default:
        throw new Error(
          `engineType: ${engine} のDatabase接続ユーザーの削除はできません`,
        );
    }
  }

  public async selectConnectionUsers(connection: any, engine: string) {
    switch (engine) {
      case 'mysql':
      case 'aurora-mysql':
        const findAllUsers4MySql = `SELECT * FROM mysql.user ;`;
        const mysqlConnectionUsers = await connection.execute(
          findAllUsers4MySql,
        );
        return mysqlConnectionUsers[0];
      default:
        throw new Error(
          `engineType: ${engine} のDatabase接続ユーザーの取得はできません`,
        );
    }
  }

  public async insertConnectionUsers(
    targetDBName: string,
    connection: any,
    engine: string,
    connectionUsers: any[],
  ) {
    switch (engine) {
      case 'mysql':
      case 'aurora-mysql':
        let insertSql4MySql: string;
        let accessUserData: any[] = [];

        switch (targetDBName) {
          case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DATABASE_NAME:
            insertSql4MySql =
              'INSERT INTO mysql.user (Host, User, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Create_user_priv, Event_priv, Trigger_priv, Create_tablespace_priv, ssl_type, ssl_cipher, x509_issuer, x509_subject, max_questions, max_updates, max_connections, max_user_connections, plugin, authentication_string, password_expired, password_last_changed, password_lifetime, account_locked, Load_from_S3_priv, Select_into_S3_priv, Invoke_lambda_priv) VALUES ?';

            let deletedPermissionUser: any[] = [];
            connectionUsers.map((user) => {
              // memo: cluster限定でinsertできなくなるので以下プロパティを削除
              delete user.Repl_slave_priv;
              delete user.Repl_client_priv;
              return deletedPermissionUser.push(user);
            });
            accessUserData = deletedPermissionUser.map((user) => {
              return Object.values(user);
            });
            break;
          case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DATABASE_NAME:
            insertSql4MySql =
              'INSERT INTO mysql.user (Host, User, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Reload_priv, Shutdown_priv, Process_priv, File_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Show_db_priv, Super_priv, Create_tmp_table_priv, Lock_tables_priv, Execute_priv, Repl_slave_priv, Repl_client_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Create_user_priv, Event_priv, Trigger_priv, Create_tablespace_priv, ssl_type, ssl_cipher, x509_issuer, x509_subject, max_questions, max_updates, max_connections, max_user_connections, plugin, authentication_string, password_expired, password_last_changed, password_lifetime, account_locked) VALUES ?';

            accessUserData = connectionUsers.map((user: any) => {
              return Object.values(user);
            });
            break;
          default:
            throw new Error(
              `targetDBName: ${targetDBName} のAccessUserInsert文は生成できません`,
            );
        }

        const filteredAccessUserData = accessUserData.filter((user: any) => {
          if (user[1] === 'rdsadmin' || user[1] === 'rdsrepladmin') {
            return false;
          }
          return true;
        });
        await connection.query(insertSql4MySql, [filteredAccessUserData]); // memo: executeだと失敗するので注意 https://github.com/sidorares/node-mysql2/issues/1239
        return await connection.query('FLUSH PRIVILEGES;'); // memo: FLUSH PRIVILEGES;を実行するにはRELOAD権限が必要
      case 'postgres':
      case 'aurora-postgresql':
        const createRoleSql = connectionUsers.join(' ').replace(/[\"]/g, '');
        return await connection.query(createRoleSql); // memo: ALL PRIVILAGESだとCREATEROLE権限はつかない
      default:
        throw new Error(
          `engineType: ${engine} のDatabase接続ユーザーの追加はできません`,
        );
    }
  }

  public async checkDatabaseConnection(
    databaseConnectInfo: IDatabase,
    engine: string,
    username: string,
    password: string,
  ) {
    const databaseRepository: DatabaseRepository = new DatabaseRepository();
    const checkConnectionInfo = {
      name: databaseConnectInfo.name,
      instanceId: databaseConnectInfo.instanceId,
      host: databaseConnectInfo.host,
      username: username,
      password: password,
    };

    const connection = await databaseRepository.connectionDatabase(
      checkConnectionInfo,
    );
    if (!connection) {
      throw new Error('Databaseへの接続に失敗しました');
    }

    let rows: any;
    const checkConnectionSql = `SELECT id FROM users LIMIT 1 ;`;
    switch (engine) {
      case 'mysql':
      case 'aurora-mysql':
        const mysqlData = connection.execute(checkConnectionSql);
        rows = mysqlData[rows];
        break;
      case 'postgres':
      case 'aurora-postgresql':
        const postgresData = connection.query(checkConnectionInfo);
        rows = postgresData.rows;
        break;
      default:
        throw new Error('想定外のEngineTypeです');
    }

    if (rows) {
      throw new Error('接続確認用ユーザーでのデータ取得に失敗しました');
    }
    return;
  }

  public checkTargetDatabase(databaseName: string) {
    const targetDatabase = [
      DATABASE_CONFIG.CLUSTER_MYSQL_DB.DB_IDENTIFIER.STAGING,
      DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DB_IDENTIFIER.STAGING,
      DATABASE_CONFIG.INSTANCE_MYSQL_DB.DB_IDENTIFIER.STAGING,
    ];
    return targetDatabase.includes(databaseName);
  }
}

domain/repository/aws/route53.ts

import {
  changeResourceRecordSets,
  postListResourceRecordSets,
  postListHostedZones,
} from '../../../infrastructure/aws/route53';
import { HostedZone } from 'aws-sdk/clients/route53';

export class Route53Repository {
  public async route53ChangeCurrentDBToRestoredDB(
    dbName: string,
    endpoint: string,
    hostedZoneId: string,
  ) {
    const resourceRecordName = this.getResourceRecordName(dbName);
    return await changeResourceRecordSets(
      hostedZoneId,
      resourceRecordName,
      endpoint,
    );
  }

  public async getHostNameFromRoute53(dbName: string, hostedZone: HostedZone) {
    const resourceRecordName = this.getResourceRecordName(dbName);
    const listResourceRecord = await postListResourceRecordSets(
      hostedZone.Id,
      resourceRecordName,
    );

    const targetRecord = listResourceRecord.ResourceRecordSets[0];
    const hostName = targetRecord?.ResourceRecords?.[0]?.Value;
    if (!hostName) {
      throw new Error('HostNameの取得に失敗しました');
    }
    return hostName;
  }

  public async getTargetHostedZone(hostedZoneName: string) {
    const hostedZones = await postListHostedZones();
    const targetHostedZone = hostedZones.HostedZones.find((hostedZone) => {
      return hostedZone.Name === hostedZoneName;
    });
    if (!targetHostedZone) {
      throw new Error('対象のHostedZoneが取得できませんでした');
    }
    return targetHostedZone;
  }
}

domain/repository/aws/ssm.ts

import { getParameter } from '../../../infrastructure/aws/ssm';
import { DATABASE_CONFIG } from '../../../config/database';

export class SsmRepository {
  public async postUsernameAndPassword4Staging(databaseName: string) {
    let usernameParameterKey: string = '';
    let passwordParameterKey: string = '';

    switch (databaseName) {
      case DATABASE_CONFIG.CLUSTER_MYSQL_DB.DATABASE_NAME:
        usernameParameterKey =
          DATABASE_CONFIG.CLUSTER_MYSQL_DB.PARAMETER_STORE.STAGING.USERNAME;
        passwordParameterKey =
          DATABASE_CONFIG.CLUSTER_MYSQL_DB.PARAMETER_STORE.STAGING.PASSWORD;
        break;
      case DATABASE_CONFIG.CLUSTER_POSTGRES_DB.DATABASE_NAME:
        usernameParameterKey =
          DATABASE_CONFIG.CLUSTER_POSTGRES_DB.PARAMETER_STORE.STAGING.USERNAME;
        passwordParameterKey =
          DATABASE_CONFIG.CLUSTER_POSTGRES_DB.PARAMETER_STORE.STAGING.PASSWORD;
        break;
      case DATABASE_CONFIG.INSTANCE_MYSQL_DB.DATABASE_NAME:
        usernameParameterKey =
          DATABASE_CONFIG.INSTANCE_MYSQL_DB.PARAMETER_STORE.STAGING.USERNAME;
        passwordParameterKey =
          DATABASE_CONFIG.INSTANCE_MYSQL_DB.PARAMETER_STORE.STAGING.PASSWORD;
        break;
      default:
        throw new Error('パラメータストアの情報を取得できません');
    }

    const usernameParameter = await getParameter(usernameParameterKey);
    const username = usernameParameter?.Parameter?.Value;
    if (!username) {
      throw new Error('接続確認用ユーザーの取得に失敗しました');
    }

    const passwordParameter = await getParameter(passwordParameterKey);
    const password = passwordParameter?.Parameter?.Value;
    if (!password) {
      throw new Error('接続確認用パスワードの取得に失敗しました');
    }

    return { username, password };
  }
}

domain/repository/spreadsheet/index.ts

import { getRows } from '../../../infrastructure/spreadsheet';

export class SpreadsheetRepository {
  public async getCreateRoleSqls(spreadsheetId: string, worksheetId: string) {
    let sqls: any[];
    const rows = await getRows(spreadsheetId, worksheetId);
    sqls = rows.filter((row) => row.nick_name).map((row) => row.SQL.replace(/\n/g, ' '));
    return sqls;
  }
}

infrastrue 레이어

infrastructure/aws/ssm.ts

import * as aws from 'aws-sdk';
const ssm = new aws.SSM({ region: 'ap-northeast-1' });

export const getParameter = async (parameterName: string) => {
  return await ssm
    .getParameter({
      Name: parameterName,
      WithDecryption: true,
    })
    .promise();
};

infrastructure/aws/route53.ts

import aws = require('aws-sdk');
const route53 = new aws.Route53({ region: 'ap-northeast-1' });

export const postListHostedZones = async () => {
  return await route53
    .listHostedZones()
    .promise();
};

export const changeResourceRecordSets = async (hostedZoneId: string, resourceRecordName: string, endpoint: string) => {
  return await route53
    .changeResourceRecordSets({
      HostedZoneId: hostedZoneId,
      ChangeBatch: {
        Changes: [
          {
            Action: 'UPSERT',
            ResourceRecordSet: {
              Name: resourceRecordName,
              ResourceRecords: [{ Value: endpoint }],
              Type: 'CNAME',
              TTL: 60,
            },
          },
        ],
      },
    })
    .promise();
};

export const postListResourceRecordSets = async (hostedZoneId: string, resourceRecordName: string) => {
  return await route53
    .listResourceRecordSets({
      HostedZoneId: hostedZoneId,
      StartRecordName: resourceRecordName,
    })
    .promise();
};

infrastructure/spreadsheet.ts

import { GoogleSpreadsheet } from 'google-spreadsheet';
import * as credentials from '../credentials.json';

export const getRows = async (spreadsheetId: string, worksheetId: string) => {
  const doc = new GoogleSpreadsheet(spreadsheetId);

  await doc.useServiceAccountAuth(credentials);
  await doc.loadInfo();

  const sheet = await doc.sheetsById[worksheetId];
  return await sheet.getRows();
};

그게 다야.

생각보다 길어요.
아직 운용이 짧기 때문에 개선할 점이 많을 것 같지만 당분간 이걸로 움직이자.
오랫동안 보살펴 주셔서 감사합니다.

직원 모집

채용 활동도 활발히 진행되고 있으니 관심 있으면 아래 기업 홈페이지를 찾아보세요.

Reference

이 문제에 관하여(공식 환경의 데이터 마스크를 제작한 후 자동으로 무대 환경과 동기화하는 메커니즘 [AWS CDK/SDK] 3~SDK 편 후반~), 우리는 이곳에서 더 많은 자료를 발견하고 링크를 클릭하여 보았다 https://zenn.dev/chycara344/articles/b794028ee3f72f

텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.

우수한 개발자 콘텐츠 발견에 전념 (Collection and Share based on the CC Protocol.)

AAuto 언어와 AMPL 표 데이터 통신 인터페이스 및 AAuto 언어는 AMPL 코드 함수를 실행합니다

useContext 후크 파트 1을 사용하여 음악 플레이어를 만드는 방법

좋은 웹페이지 즐겨찾기

개발자 우수 사이트 수집

개발자가 알아야 할 필수 사이트 100선 추천 우리는 당신을 위해 100개의 자주 사용하는 개발자 학습 사이트를 정리했습니다